feat(tvix/tools/crunch-v2): init

Browse source

This is a tool for ingesting subsets of cache.nixos.org into its own flattened castore format.
Currently, produced chunks are not preserved, and this purely serves as a way of measuring
compression/deduplication ratios for various chunking and compression parameters.

Change-Id: I3983af02a66f7837d76874ee0fc8b2fab62ac17e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10486
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>

...

This commit is contained in:

edef 2024-01-17 16:04:03 +00:00

parent e0a1c03b24

commit 4f22203a3a

12 changed files with 15022 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

3041

tvix/tools/crunch-v2/Cargo.lock generated Normal file

View file

File diff suppressed because it is too large Load diff