feat(tvix/castore/directory): add bigtable backend

This adds a Directory service using
https://cloud.google.com/bigtable/docs/ as a K/V store.

Directory (closures) are put in individual keys.

We don't do any bucketed upload of directory closures (yet), as castore/
fs does query individually, does not request recursively (and buffers).
This will be addressed by store composition at some point.

Change-Id: I7fada45bf386a78b7ec93be38c5f03879a2a6e22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11212
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
This commit is contained in:
Florian Klink 2024-03-19 12:12:03 +02:00 committed by clbot
parent 84ad8a0bbd
commit 17849c5c00
12 changed files with 2767 additions and 22 deletions

View file

@ -4,5 +4,9 @@ depot.tvix.crates.workspaceMembers.tvix-castore.build.override {
runTests = true;
testPreRun = ''
export SSL_CERT_FILE=${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt;
export PATH="$PATH:${pkgs.lib.makeBinPath [pkgs.cbtemulator pkgs.google-cloud-bigtable-tool]}"
'';
# enable some optional features.
features = [ "default" "cloud" ];
}