From 0814eda06b471caac95b0b137c04bfef2f0bc754 Mon Sep 17 00:00:00 2001 From: zimbatm Date: Thu, 17 Oct 2024 11:10:14 +0000 Subject: [PATCH] docs(users/edef/fetchroots): add a README for other users Change-Id: Ibce9121fbd209e713ce04271e285f70e2af9867e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12646 Tested-by: BuildkiteCI Reviewed-by: edef Reviewed-by: flokli --- users/edef/fetchroots/README.md | 36 +++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 users/edef/fetchroots/README.md diff --git a/users/edef/fetchroots/README.md b/users/edef/fetchroots/README.md new file mode 100644 index 000000000..f20cfbf28 --- /dev/null +++ b/users/edef/fetchroots/README.md @@ -0,0 +1,36 @@ +# fetchroots + +> This tool is part of a suite of tools built to manage cache.nixos.org. + +This tool's purpose is to build an index of all the GC roots from the +channels.nixos.org releases. The result is then combined with other tools. + +It does this by: +1. Listing all the release files in the bucket. +2. Getting the data for each of the release. +3. Putting them in a local parquet file. + +## Getting started + +In order to run this, you'll need AWS SSO credentials from the NixOS Infra team. + +Get the creds from https://nixos.awsapps.com/start/ -> LBNixOS_Dev_PDX -> AWSReadOnlyAccess. + +Run `mg run`, you should see a progress bar. + +Congrats, you now have a `roots.parquet` file. You can now load it with python polars-rs or clickhouse. + +## `roots.parquet` file format + + * `key` (`String`): the release, eg `nixos/22.11-small/nixos-22.11.513.563dc6476b8` + * `timestamp` (`DateTime`): the timestamp of the GC roots file for this release + * `store_path_hash` (`List[Binary]`): hash part of the store paths rooted by this release + +## Development + +When the Cargo.lock changes, run `mg run //tools:crate2nix-generate`. + +To build the project, run `mg build`. + +To get a dev environment, run `nix-shell -p cargo`. +