feat(tvix/castore/directory): add bigtable backend

This adds a Directory service using
https://cloud.google.com/bigtable/docs/ as a K/V store.

Directory (closures) are put in individual keys.

We don't do any bucketed upload of directory closures (yet), as castore/
fs does query individually, does not request recursively (and buffers).
This will be addressed by store composition at some point.

Change-Id: I7fada45bf386a78b7ec93be38c5f03879a2a6e22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11212
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
This commit is contained in:
Florian Klink 2024-03-19 12:12:03 +02:00 committed by clbot
parent 84ad8a0bbd
commit 17849c5c00
12 changed files with 2767 additions and 22 deletions

View file

@ -43,6 +43,10 @@ let
nativeBuildInputs = protobufDep prev;
};
prost-wkt-types = prev: {
nativeBuildInputs = protobufDep prev;
};
tonic-reflection = prev: {
nativeBuildInputs = protobufDep prev;
};
@ -84,6 +88,7 @@ let
(crateName:
(lib.nameValuePair "${crateName}-${crates.internal.crates.${crateName}.version}" crates.internal.crates.${crateName}.src.outputHash)
) [
"bigtable_rs"
"test-generator"
"wu-manber"
]);