We don't need to be as verbose with the types, we can use Ok::<_,Status>
inside the async block.
Change-Id: Ibc92a9f7bdbc5031806b7eca82a01e2c7cb48c0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13264
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
We don't do any cloning of self in here.
Change-Id: Id6d4aed25244f5a18eb663ad8686fab4c6aaf20b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13263
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Before uploading a blob, a client usually checks whether a blob exists
via the stat method. We return a NotFound if it's not present.
Our TraceLayer currently considers this an error, logging a message.
This gets very noisy. Exclude InvalidArgument and NotFound, this is
expected behaviour.
Change-Id: I7da22a41fc6f3b8d422521a01530fc9e638e33ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13257
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
This means we don't need to clone self in all implementations using
SimplePutter.
Change-Id: I11347028801c43c5cc9920c0a9edf47627096bce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13262
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
We don't need an owned version of the database, a reference is
sufficient.
Change-Id: I84d10ad39c1db28f4f1dda26514c04418db48a27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13261
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
We can share the base_path with the DirectoryService.
We cannot share self.object_store, as object_store::buffered::BufWriter
wants an Arc<dyn DirectoryService>.
Change-Id: Icfe9d256774de48bb3b4d8afd32620a8188ecfdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13260
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
We can require DirectoryPutters to not outlive the DirectoryServices
they originated from.
This allows borrowing some data from there.
Change-Id: I82a4c17b06b3b0ae5cf84d35561f1aae049a2cdc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13259
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
This does import the contents into the store, so it should belong in there.
While moving, I also noticed the code copying to the BlobService can be
shared with the one used when importing a blob via import helper.
It was a bit hidden - due to the contents being available as a bytes, we
used a Cursor and wrote it with tokio::io::copy.
However, Cursor implements both AsyncRead and Read, so we can factor out
the copying code into a copy_to_blobservice helper function and use it
in both places.
The output name being wrong error kind arguably is still a bit misplaced
here, but that's left for later.
Change-Id: Iec3c422c12270ee111f864d2b78c0861f78edfa4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13254
Reviewed-by: Domen Kožar <domen@cachix.org>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
We cannot just escape the square brackets as rustdoc suggests, because
clap outputs the docstring verbatim, without any escaping to the user.
Change-Id: I7ce7932e35ee1b01bccbc142a7bdc7af9ce7e51e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13256
Tested-by: BuildkiteCI
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Domen Kožar <domen@cachix.org>
This modifies the EvaluationBuilder, not Evaluation.
Also add some missing builtins from the docstrings.
Change-Id: Ibf76b496874158fb85b21e52e7c3561e66cffaba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13253
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Domen Kožar <domen@cachix.org>
builtins.filterSource and builtins.path itself are very short these
days, and use the same underlying (private) helper function.
Describing builtins.filterSource with builtins.path won't lead to less
code, so this TODO can be dropped.
Change-Id: I5457949136e19204b32c8182b6c89e66cc2183e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13252
Reviewed-by: Domen Kožar <domen@cachix.org>
Tested-by: BuildkiteCI
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
We only pull in tvix_eval in dev dependencies, so this cannot link to
the type.
Change-Id: I26948a32567e2dc8e846601c05b820d9aa14605f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13250
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Reviewed-by: Domen Kožar <domen@cachix.org>
Tested-by: BuildkiteCI
There were some broken references. Also, during the split of service
urls to castore and store parts we forgot to update the docstring.
Change-Id: Ifc2b3a2c49c9e4d31f1b210dbfe4afcad8a8d5bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13249
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Domen Kožar <domen@cachix.org>
cl/13156 did move the castore-specific URLs into their own struct,
having the store one include the castore-specific struct (flattened).
However, both structs had the same name, so using these structs causes
clap to panic at runtime, as two groups with the same id were present.
Fix this, by adding names to (all) structs.
Reported-By: Domen Kožar <domen@cachix.org>
Change-Id: I34064f7fb2fbbc19d836e1486ad84b52548d2ee1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13247
Tested-by: BuildkiteCI
Reviewed-by: Domen Kožar <domen@cachix.org>
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Fix a bug introduced in 2daa483249
When the -I or --extra-nix-path option is not passed, the cli will
interpret the contents of the NIX_PATH environment variable twice:
once as part of the env attribute of clap's derive macro, and again in a
custom nix_path function. The custom function implements the appending
behavior nix users likely expect (see commit mentioned above), so the
env attribute is dropped. The downside is that NIX_PATH is no longer
rendered in the cli's help text. We can reclaim this behavior if we want
to, but that's left for later.
Change-Id: I2c0939d9453ea1af73e9ec15e45b25bf035a7493
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13207
Autosubmit: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This has been down for a while, and people have been asking me why we have a
dead link on the website.
Change-Id: I7b2bc8dd7a627678fbb8d48e86ad6eb6f1c73aa2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13222
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This error type is a bit less annoying to use in other errors, as it
doesn't hang onto bytes from the input.
Change-Id: I4b969717d443f5e04dd2086baeb946a98c4f9d7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13206
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
This makes it possible for external consumers to deal with this error.
Change-Id: I70b16efb99c6a98c75b1672ed0618ac963f5f79d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13194
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
This adds code allowing to parse (and emit) the "internal-json" log
format that Nixcpp emits.
Move the Verbosity struct from worker_protocol.rs out into
src/log/mod.rs, it's not only used there.
Change-Id: Ia35e739c83e06080a677b576616bdd0d4a821cf4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13167
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: edef <edef@edef.eu>
See 72dd5818b7/CHANGELOG.md
for the nom changelog.
Most notably, there's now a .parse() to be added:
`combinator(arg)(input)` -> `combinator(arg).parse(input)`
There also doesn't need to be a tuple combinator (it's implemented on
tuples directly).
This also refactors the string / byte field parsing parts, to make them
more concise.
Change-Id: I9e8a3cedd07d6705be391898eb6a486fb8164069
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13193
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: Brian Olsen <me@griff.name>
Update the tvix cli's -I option so that it aligns more closely with
nix's behavior: prepending entries to the list of lookup paths provided
by the NIX_PATH environment variable. Before this commit, using the -I
option would instead override and ignore the NIX_PATH variable.
Additionally, update the option's long name and help text to match the
new behavior.
While the tvix cli's interface does not appear to be attempting to mimic
nix exactly, I think this particular case of the -I option's diverging
behavior will inevitably surprise users because it's name, presumably
short for "include" and being similar to gcc's flag, evokes additivity.
The prior implementation hinted at this difference with the help text
and the long name, --nix-search-path, but I still suspect users will be
confused on first usage (at least I was). If we're willing to pay the
maintenance costs of additional code, we can avoid this and provide a
slightly smoother user experience.
Changes were tested by buiding the tvix cli, adding it to the PATH, and
executing simple tests as in the following bash script
mg build //tvix/cli
PATH="$PWD/result/bin:$PATH"
one=$(mktemp) && echo "=> $one :: path" > "$one"
two=$(mktemp) && echo "=> $two :: path" > "$two"
dir1=$(mktemp -d) && file1="$dir1/file1" && echo "=> $file1 :: path" > "$file1"
dir2=$(mktemp -d) && file2="$dir2/file2" && echo "=> $file2 :: path" > "$file2"
# NIX_PATH works with a single non-prefixed lookup path.
NIX_PATH="$dir1" tvix -E "<file1>" | cmp - "$file1"
# NIX_PATH works with multiple non-prefixed lookup paths.
NIX_PATH="$dir1:$dir2" tvix -E "<file2>" | cmp - "$file2"
# NIX_PATH works with a single prefixed lookup path.
NIX_PATH="one=$one" tvix -E "<one>" | cmp - "$one"
# NIX_PATH works with multiple prefixed lookup paths.
NIX_PATH="one=$one:two=$two" tvix -E "<one>" | cmp - "$one"
NIX_PATH="one=$one:two=$two" tvix -E "<two>" | cmp - "$two"
# NIX_PATH first entry takes precedence.
NIX_PATH="one=$one:one=$two" tvix -E "<one>" | cmp - "$one"
# The -I option works with a single non-prefixed lookup path.
tvix -I "$dir1" -E "<file1>" | cmp - "$file1"
# The -I option works with multiple non-prefixed lookup paths.
tvix -I "$dir1" -I "$dir2" -E "<file2>" | cmp - "$file2"
# The -I option works with a single prefixed lookup path.
tvix -I "one=$one" -E "<one>" | cmp - "$one"
# The --extra-nix-path option works with a single prefixed lookup path.
tvix --extra-nix-path "one=$one" -E "<one>" | cmp - "$one"
# The -I options works when passed multiple times with prefixed lookup paths.
tvix -I "one=$one" -I "two=$two" -E "<one>" | cmp - "$one"
tvix -I "one=$one" -I "two=$two" -E "<two>" | cmp - "$two"
# The first -I option takes precedence.
tvix -I "one=$one" -I "one=$two" -E "<one>" | cmp - "$one"
# Both NIX_PATH and the -I option work together and are additive.
NIX_PATH="one=$one" tvix -I "two=$two" -E "<one>" | cmp - "$one"
NIX_PATH="one=$one" tvix -I "two=$two" -E "<two>" | cmp - "$two"
# The -I option takes precedence over NIX_PATH.
NIX_PATH="one=$one" tvix -I "one=$two" -E "<one>" | cmp - "$two"
rm "$one"
rm "$two"
rm "$file1" && rmdir "$dir1"
rm "$file2" && rmdir "$dir2"
The above script assumes it's being run from inside the depot.
Change-Id: I153e6de57939c0eeca1f9e479d807862ab69b2de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13189
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Fix "environent" typo in the Builder Protocol document. Remove "todo"
label from the nixVersion builtin because it appears to have been
implemented in 7fe1331570
Change-Id: I64aaa7cc93c8c6978cedd3bfacf756dacb4c7fc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13188
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
* //tvix/*-go: regenerate protobuf files
* //tvix/boot:
- Explicitly set compression method of mkBinaryCache which has
made this configurable and (at the same time) changed the default.
- Adjust to change of extension of the nar files from .nar.xz to
plain .xz.
Change-Id: Ie79ea8e0ac8fe04ae01f5558bffca93e9314f56d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13174
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
These days highlight.js in mdbook includes syntax highlighting for Nix
out of the box, there's no more need to ship our own variant of it.
Change-Id: Idd453c2efe97707ae861de9fa08c9dab71ea7876
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13178
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: adisbladis <adisbladis@gmail.com>
Tested-by: BuildkiteCI
Currently ingest_entries panics. This makes it hard to use
ingest_entries in parallel with other processes. When another process
runs into an error and wants to return so, while ingest_entries has
already started and has no entries yet, we're forced to panic.
With this change ingest_entries will return an error when there are no
entries, thus allowing the user to handle the errors as they please.
Change-Id: I78b85bf18f52af8c157d6bedad6019fd4398250a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13146
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Bob van der Linden <bobvanderlinden@gmail.com>
Reviewed-by: Bob van der Linden <bobvanderlinden@gmail.com>
Tested-by: BuildkiteCI
We're not saying 2.3 here anymore in all cases, but use the version
specified in the `nix_version` function argument.
Change-Id: I86c34f97eea4141b8431de7c6a784750d6b7c2cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13166
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
This provides a shell containing the necessary packages to run the
integration tests too.
//tvix:shell stays the same.
Change-Id: Iac7977cbb7dc085da698993be9baca662d09ff5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13164
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Instead of collecting all rows into a Vec, use the stream_rows function,
and massage Ok values into the right shape.
This avoids collecting everything into memory.
Change-Id: I2f03434c7e1e25ee8395c69685b16100aec36f89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13163
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Since cl/12661 this doesn't need to be Arc'd.
Change-Id: I3c14c45df7815867ddea6f461e81649772648962
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13155
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
make the Display impl of B3Digest use the `blake3-` prefix rather than
`b3:`, which is more close to the SRI hashes we use in NixHash.
Also use Span::current().record() to record the `blob.digest` field (and
only when the digest has the right lenght). It doesn't make sense to
brand it as blake3 before anyways, and allows us to get rid of the
manual implementation.
Change-Id: Iee349557ef4761807af1fb3942387de1942ab12b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13162
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
This is probably a nicer default, and also aligns more with Serialize.
Change-Id: I3a8f18607c305dbf9fb93a2e2dd02ad07cd99f24
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13161
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Included changes:
* targeted primary NixOS channel back at `nixos-unstable`
* //3p/chicago95: disable new broken symlink check
* //tvix: omit unnecessary lifetime clippy recommends
* //users/sterni/blipqn: wait a bit for data to arrive in test so it
succeeds under load.
* //fun/paroxysm: force pkg-config flag of pq-sys by adding a bogus
dependency on it. Otherwise, pq-sys will try to use pg_config
which does not work correctly in pkgs.libpq at the moment.
* //users/flokli/keyboards/dilemma: disable temporarily
Change-Id: I6d53bd7bca6886f3457e1f41505e97314f4cd191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13119
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
This got disabled in 5f17df8548 without
much investigation.
After some more digging, I found the root cause -
173de320197ba97f2daf46b7d229055cd3732df4 in nixpkgs changed the files
produced by `mkBinaryCache` to use FileHash, not NarHash to name NAR
files.
While this is more correct and consistent with how it's produced by Nix,
it unfortunately means our uploads will now get rejected.
We didn't notice this, as curl (and rush) were not configured to bail
out - which now is the case again.
Deal with the change, by decompressing twice, once to get the NarHash
again, the second time to feed the data to curl.
Change-Id: Ic1deb470801250e7967083ac78cd46be39f3caa3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13122
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
The TVL overlay can lead to diverging store paths, e.g. starting with
CL/13119, firefox depends on some FODs that in turn depend on Nix. Thus
the TVL overlay (with Nix 2.3) will come up with a different drvPath
than just using (import pkgs.path {}).
To fix this, just automatically use the same arguments to nixpkgs as
Tvix for expectedPath.
Change-Id: I77d39df9092442a49c8ee94b012dec046ff6a07e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13147
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Included changes/fixes:
* bumped all `wasm-bindgen` usages again
* regenerated protobuf files
* keycloak terraform provider has been migrated to new name
This also included a state migration in the bucket, which I've already
performed.
* tvix/boot: disable tests that are broken in CI
* users/aspen/yeren: avoid upgrading kernel to 6.12
digimend depends on a fix: https://github.com/NixOS/nixpkgs/pull/378830/
Change-Id: I657dcf5c4d0d08f231bfe30e37c8062bfcfaaa32
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13098
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
I've checked the new links manually.
- //web/tvl: changed the target path of links to the tvix docs since
they were moved in r/2378.
- //users/aspen/resume: Tvix is no longer //third_party/nix.
Change-Id: I419bae1a46bdccc7baa7327215aa2368ffc0f01c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13043
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: fogti <fogti+devel@ytrizja.de>
Reviewed-by: tazjin <tazjin@tvl.su>
