feat(tvix/tools/crunch-v2): init
This is a tool for ingesting subsets of cache.nixos.org into its own flattened castore format. Currently, produced chunks are not preserved, and this purely serves as a way of measuring compression/deduplication ratios for various chunking and compression parameters. Change-Id: I3983af02a66f7837d76874ee0fc8b2fab62ac17e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10486 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
This commit is contained in:
parent
e0a1c03b24
commit
4f22203a3a
12 changed files with 15022 additions and 0 deletions
10
tvix/tools/crunch-v2/src/lib.rs
Normal file
10
tvix/tools/crunch-v2/src/lib.rs
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
use lazy_static::lazy_static;
|
||||
|
||||
pub mod proto {
|
||||
include!(concat!(env!("OUT_DIR"), "/tvix.flatstore.v1.rs"));
|
||||
}
|
||||
|
||||
lazy_static! {
|
||||
static ref DB: sled::Db = sled::open("crunch.db").unwrap();
|
||||
pub static ref FILES: sled::Tree = DB.open_tree("files").unwrap();
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue