We don't need to be as verbose with the types, we can use Ok::<_,Status>
inside the async block.
Change-Id: Ibc92a9f7bdbc5031806b7eca82a01e2c7cb48c0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13264
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
We don't do any cloning of self in here.
Change-Id: Id6d4aed25244f5a18eb663ad8686fab4c6aaf20b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13263
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This means we don't need to clone self in all implementations using
SimplePutter.
Change-Id: I11347028801c43c5cc9920c0a9edf47627096bce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13262
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
We don't need an owned version of the database, a reference is
sufficient.
Change-Id: I84d10ad39c1db28f4f1dda26514c04418db48a27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13261
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
We can share the base_path with the DirectoryService.
We cannot share self.object_store, as object_store::buffered::BufWriter
wants an Arc<dyn DirectoryService>.
Change-Id: Icfe9d256774de48bb3b4d8afd32620a8188ecfdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13260
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
We can require DirectoryPutters to not outlive the DirectoryServices
they originated from.
This allows borrowing some data from there.
Change-Id: I82a4c17b06b3b0ae5cf84d35561f1aae049a2cdc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13259
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
There were some broken references. Also, during the split of service
urls to castore and store parts we forgot to update the docstring.
Change-Id: Ifc2b3a2c49c9e4d31f1b210dbfe4afcad8a8d5bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13249
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Domen Kožar <domen@cachix.org>
cl/13156 did move the castore-specific URLs into their own struct,
having the store one include the castore-specific struct (flattened).
However, both structs had the same name, so using these structs causes
clap to panic at runtime, as two groups with the same id were present.
Fix this, by adding names to (all) structs.
Reported-By: Domen Kožar <domen@cachix.org>
Change-Id: I34064f7fb2fbbc19d836e1486ad84b52548d2ee1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13247
Tested-by: BuildkiteCI
Reviewed-by: Domen Kožar <domen@cachix.org>
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Currently ingest_entries panics. This makes it hard to use
ingest_entries in parallel with other processes. When another process
runs into an error and wants to return so, while ingest_entries has
already started and has no entries yet, we're forced to panic.
With this change ingest_entries will return an error when there are no
entries, thus allowing the user to handle the errors as they please.
Change-Id: I78b85bf18f52af8c157d6bedad6019fd4398250a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13146
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Bob van der Linden <bobvanderlinden@gmail.com>
Reviewed-by: Bob van der Linden <bobvanderlinden@gmail.com>
Tested-by: BuildkiteCI
Since cl/12661 this doesn't need to be Arc'd.
Change-Id: I3c14c45df7815867ddea6f461e81649772648962
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13155
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
make the Display impl of B3Digest use the `blake3-` prefix rather than
`b3:`, which is more close to the SRI hashes we use in NixHash.
Also use Span::current().record() to record the `blob.digest` field (and
only when the digest has the right lenght). It doesn't make sense to
brand it as blake3 before anyways, and allows us to get rid of the
manual implementation.
Change-Id: Iee349557ef4761807af1fb3942387de1942ab12b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13162
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Included changes:
* targeted primary NixOS channel back at `nixos-unstable`
* //3p/chicago95: disable new broken symlink check
* //tvix: omit unnecessary lifetime clippy recommends
* //users/sterni/blipqn: wait a bit for data to arrive in test so it
succeeds under load.
* //fun/paroxysm: force pkg-config flag of pq-sys by adding a bogus
dependency on it. Otherwise, pq-sys will try to use pg_config
which does not work correctly in pkgs.libpq at the moment.
* //users/flokli/keyboards/dilemma: disable temporarily
Change-Id: I6d53bd7bca6886f3457e1f41505e97314f4cd191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13119
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Last one of the year! С наступающим)
Fixes:
* users/wpcarro: remove use-package from emacs packages (it has been built-in
for a while now)
* users/sterni: the same thing
* users/aspen: remove `coz`, forwardport `gdmap` from stable
* users/flokli: dropped corneish_zen firmware from CI
This firmware depends on a non-reproducible FOD which, when updated, causes
build failures. We have worked around this repeatedly, but it needs to be
fixed properly.
* tvix: regenerate Go protobufs
* tvix: address new clippy lints
* tvix/{castore,store,build}-go: update grpc/protobuf libraries
* tvix/eval: formatting fixes
* 3p/overlays/tvl: work around GCC 14 -Werrors
Change-Id: Ice5948ca7780192fb7d2abc6a48971fb875f03c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12933
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
We don't need to use a git checkout here anymore.
Change-Id: I99641a4908f39799c8be6a1610ae458ca6fdc5ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12931
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Use dotted keys where we'd else have maps with a single key.
Change-Id: I9389e0fedddad1cf65f870a3a68415a0defaa259
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12929
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Extend the docstrings of `add_default_services`, and add one for
`addrs_to_configs` as well as the module-wide one at
`tvix_store::composition`.
Change-Id: Ie9b449988eb210cd65b19b174094bbe0c4af2fd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12748
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
tvix-castore already supports composition without any additional feature
flags, the only thing that can be explicitly enabled is referring to
other stores via an anonymous url. Rename that feature flag to
"xp-composition-url-refs".
tvix-store effectively only controls the CLI surface, so rename this to
"xp-composition-cli".
The "store" in the feature name was dropped, as it's already apparent
from being in the tvix-[ca]store crate.
Change-Id: I1175dc6280cbba4cbcdfb7fd4b35fce713b45fc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12747
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
This becomes the root of the composition. `default` implies we can
directly access anything else, which we cannot. `root` makes this more
understandable, and it's all internal only anyways.
Change-Id: I297511bc05a7c32c59510b9d192b40d1bd937b5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12746
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Currently it is not possible to distinguish between tracing of the same
*Service type whenever there are multiple of them. Now the instance_name
of ServiceBuilder is passed into the *Service and used in the existing
instrument as the `instance_name` field.
Places that did not already have a instance_name in its context use
`"default"`. In tests I used `"test"`.
Change-Id: Ia20bf2a7bb849a781e370d087ba7ddb3be79f654
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12739
Tested-by: BuildkiteCI
Autosubmit: Bob van der Linden <bobvanderlinden@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
fuse_backend_rs::api::filesystem::Layer is not exposed for non-Linux, and
feature-flagged on virtiofs, so only implement the trait for these cases.
Change-Id: Id8455dc5be502f8375836ba04288d50c59d69d89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12693
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This pushes generating spans with pb_show up to the caller.
They usually have more context on how to present things, if at all.
Change-Id: Icfcaa64a8a57dce50c0261f2d06e7c051e3946c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12657
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
We have two places where we parse protos and want their names to be
empty:
- Receiving a root node in a nar-bridge NAR request
- Processing the CalculateNAR gRPC call
We don't have any place where we want to keep a name as bytes::Bytes
around, yet we used the `into_name_bytes_and_node` method.
It was also a bit wrongly named - it wasn't very clear the name was
not validated, and that the function may fail.
This moves the "splitting off the name as bytes::Bytes" part into a
private helper, only leaving the `try_into_name_and_node` and
`try_into_anonymous_node` methods around.
Change-Id: I2c7fd9871d49ec67450d7efa6a30d96197fb319c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12664
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This implements BS, DS, PS for Box'ed or Arc'ed variants of it with less
code, and less potential to accidentially forget to proxy default trait
methods for blanked impls, as fixed in cl/12658.
Change-Id: If2cdbb563a73792038ebe7bff45d6f880214855b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12661
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.
Change-Id: Ia0370ca46cb1c6122a452b1d117160536b632c7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12612
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This became obsolete, since the introduction of a stricter `Directory`
struct invalid names cannot be represented anymore.
Change-Id: I9e4b1b6cca01831d0a9735f58d8a1f59ac18676b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12615
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This switches the PathInfoService trait from using the proto-derived
PathInfo struct to a more restrictive struct, and updates all
implementations to use it.
It removes a lot of the previous conversion and checks, as invalid
states became nonrepresentable, and validations are expressed on the
type level.
PathInfoService implementations consuming protobuf need to convert and
do the verification internally, and can only return the strongly typed
variant.
The nix_compat::narinfo::NarInfo conversions for the proto PathInfo
are removed, we only keep a version showing a NarInfo representation for
the strong struct.
Converting back to a PathInfo requires the root node now, but is
otherwise trivial, so left to the users.
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I6fdfdb44063efebb44a8f0097b6b81a828717e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12588
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This allows specifying an url in place of a named reference to another
composition entry, if the castore crate has been compiled with the
xp-store-composition feature.
Example: `--directory-service-addr cache://?near=memory://&far=memory://`
This would be equivalent to the instantiation via toml file:
```toml
[memory1]
type = "memory"
[memory2]
type = "memory"
[default]
type = "cache"
near = "memory1"
far = "memory2"
```
Note that each anonymous url causes a distinct instance to be created.
Change-Id: Iee5a07a94b063b5e767c704d9cad0114fa843164
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12146
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We wanted to enable this earlier but the log level of many of the
messages in redb were too high, they've now been downgraded, so we can
enable logs from redb.
Context on the fix and release:
- https://github.com/cberner/redb/pull/828
- https://github.com/cberner/redb/releases/tag/v2.1.2
Change-Id: I8635e8a0bcb01a7d0b580387ac9134ccdd0205f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12568
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Over the past couple of months we've been using redb instead of sled as
the default filesystem-based database in PS and DS. I am confident that
we can get rid of sled completely now, and just keep redb.
Change-Id: I11fa1e4453e280253855f8eade990b37eb6965ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12567
Reviewed-by: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Previously, the overlap calculation would underflow when
the pattern is empty.
Change-Id: I1f6bf49fafc4b8183a3a5e5e491a5a5bfc41ca97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12558
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
This changes the only actual state the ReferenceScanner has to use atomic bools, so it no longer requires a mutable borrow for .scan(). This allows passing an immutable borrow of a reference scanner to multiple threads which might be ingesting blobs in parallel, and using
them in the ReferenceReader or calling .scan() there.
Change-Id: Id5c30bcebb06bf15eae8c4451d70eb806cab722e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12528
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This release includes support for overlay filesystems
Change-Id: I946cbf346df045209afaea2d720bb57fc2f2659f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12538
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
This is required to add the optional refscanner parameter to the
ingest functions.
Change-Id: Ib40a7287cf857eb55e31e0df309a79474fefb518
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12527
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This otherwise spams the logs quite a bit, for example when uploading
to nar-bridge with the new check from cl/12497.
Change-Id: Idc2bcc513caea6fae38ae04489e4e3ee7be64bce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12510
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Nar-bridge tried to parse the name of the protobuf node encoded in the
URL into a PathComponent but this name was empty, leading to an error
when the user tried to retrieve the nar file.
This was an oversight from the conversion to stricter types (some of the
CLs in the serious containing cl/12217).
We need a version converting a protobuf without a name to our stricter
types, but an empty PathComponent cannot be constructed.
So we need a into_name_and_node() version that returns the name as
Bytes, not PathComponent.
Change-Id: I2996cdd2e0107133e502748947298f512f1cc521
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12504
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
ConcurrentBlobUploader buffers small blobs in memory, and then uploads
them to the BlobService in the background.
In these cases, we know the hash of the whole blob, so we could check if
it exists first before, uploading it.
We were however not, and this caused rate limiting issues in GCS, as it
has an update limit of one write per second on the same key, which we
ran into especially frequently with the empty blob.
This reduces the amount of writes of the same blob considerably.
In the future, we might be able to drop this, as our chunked blob
uploading protocol gets smarter and covers these cases.
Change-Id: Icf482df815812f80a0b65cec0426f8e686308abb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12497
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
