maurice/snix
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
snix/contrib/fetchroots
History
Brian Olsen e5177ae930 feat(nix-compat): Use serde_with for HashAlgo serialization 
We already have serde_with in the workspace and using it cuts down
on code lines (very slightly).

This also adds a FromStr implementation for HashAlgo in addition to
the existing TryFrom<&str>.

Change-Id: I1b525d7391f3881d54cef0c74d22198ad394f47d
Reviewed-on: https://cl.snix.dev/c/snix/+/30652
Reviewed-by: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Autosubmit: Brian Olsen <brian@maven-group.org>
2025-08-05 22:23:54 +00:00
..
src chore(contrib): upgrade to 2024 edition 2025-07-04 18:19:41 +00:00
.gitignore chore(users/edef): move to contrib 2025-03-17 17:15:08 +00:00
Cargo.lock feat(nix-compat): Add serde feature flag 2025-08-05 22:04:21 +00:00
Cargo.nix feat(nix-compat): Use serde_with for HashAlgo serialization 2025-08-05 22:23:54 +00:00
Cargo.toml chore(contrib): upgrade to 2024 edition 2025-07-04 18:19:41 +00:00
default.nix chore(users/edef): move to contrib 2025-03-17 17:15:08 +00:00
README.md chore(users/edef): move to contrib 2025-03-17 17:15:08 +00:00

README.md

fetchroots

This tool is part of a suite of tools built to manage cache.nixos.org.

This tool's purpose is to build an index of all the GC roots from the channels.nixos.org releases. The result is then combined with other tools.

It does this by:

  1. Listing all the release files in the bucket.
  2. Getting the data for each of the release.
  3. Putting them in a local parquet file.

Getting started

In order to run this, you'll need AWS SSO credentials from the NixOS Infra team.

Get the creds from https://nixos.awsapps.com/start/ -> LBNixOS_Dev_PDX -> AWSReadOnlyAccess.

Run mg run, you should see a progress bar.

Congrats, you now have a roots.parquet file. You can now load it with python polars-rs or clickhouse.

roots.parquet file format

  • key (String): the release, eg nixos/22.11-small/nixos-22.11.513.563dc6476b8
  • timestamp (DateTime): the timestamp of the GC roots file for this release
  • store_path_hash (List[Binary]): hash part of the store paths rooted by this release

Development

When the Cargo.lock changes, run mg run //tools:crate2nix-generate.

To build the project, run mg build.

To get a dev environment, run nix-shell -p cargo.