maurice/snix
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(tvix/castore): ingest filesystem entries in parallel

Browse source 
Rather than carrying around an Future in the IngestionEntry::Regular,
simply carry the plain B3Digest.

Code reading through a non-seekable data stream has no choice but to
read and upload blobs immediately, and code seeking through something
seekable (like a filesystem) probably knows better what concurrency to
pick when ingesting, rather than the consuming side.

(Our only) one of these seekable source implementations is now doing
exactly that. We produce a stream of futures, and then use
[StreamExt::buffered] to process more than one, concurrently.

We still keep the same order, to avoid shuffling things and violating
the stream order.

This also cleans up walk_path_for_ingestion in castore/import, as well
as ingest_dir_entries in glue/tvix_store_io.

Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This commit is contained in:
Florian Klink 2024-04-20 19:32:30 +03:00 • committed by clbot
parent 01239a4f6f
commit 5fc403587f
4 changed files with 53 additions and 82 deletions
Download patch file Download diff file Expand all files Collapse all files

13
tvix/glue/src/builtins/import.rs
View file

@ -2,6 +2,7 @@
use crate::builtins::errors::ImportError;
use std::path::Path;
use tvix_castore::import::ingest_entries;
use tvix_eval::{
builtin_macros::builtins,
generators::{self, GenCo},
@ -84,15 +85,19 @@ async fn filtered_ingest(
entries.push(entry);
}
let entries_iter = entries.into_iter().rev().map(Ok);
let dir_entries = entries.into_iter().rev().map(Ok);
state.tokio_handle.block_on(async {
state
.ingest_dir_entries(entries_iter, path)
let entries = tvix_castore::import::fs::dir_entries_to_ingestion_stream(
&state.blob_service,
dir_entries,
path,
);
ingest_entries(&state.directory_service, entries)
.await
.map_err(|err| ErrorKind::IO {
path: Some(path.to_path_buf()),
error: err.into(),
error: Rc::new(err.into()),
})
})
}