Part of #114
The lock at `nix-compat/src/nix_daemon/handler.rs:186` gets released later, but
this doesnt seem to change behavior.
Change-Id: If871a090791740d98bca7a043e86af6a9b0c5463
Reviewed-on: https://cl.snix.dev/c/snix/+/30590
Autosubmit: Bence Nemes <nemes.bence1@gmail.com>
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
Part of #114
This edition change doesn't affect these crates except for formatting.
Change-Id: I441c967b6803f1f839379c2ec13dee3f0958fb05
Reviewed-on: https://cl.snix.dev/c/snix/+/30593
Autosubmit: Bence Nemes <nemes.bence1@gmail.com>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Part of #114
This edition change doesn't affect this crate except for formatting.
Change-Id: I8e2650f6de776658a14ee5a11bc7a63ad57fae89
Reviewed-on: https://cl.snix.dev/c/snix/+/30591
Reviewed-by: Florian Klink <flokli@flokli.de>
Autosubmit: Bence Nemes <nemes.bence1@gmail.com>
Tested-by: besadii
Use entry.file_type() instead of entry.metadata().file_type() to avoid
unnecessary stat syscalls. The file_type() method uses the cached d_type
field from the readdir() syscall when available, eliminating the need
for additional filesystem access.
This matches the behavior described in https://github.com/NixOS/nix/pull/13412
where readDir should only read directory entries without stat'ing each file.
Change-Id: I8983f84a7cff28b2c892d5829b9880ed22be4389
Reviewed-on: https://cl.snix.dev/c/snix/+/30596
Reviewed-by: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Part of #114
This edition change doesn't affect this crate except for formatting.
Change-Id: Ia7adca8a62e3cb7cbc1187c539907be45a05ab4b
Reviewed-on: https://cl.snix.dev/c/snix/+/30589
Autosubmit: Bence Nemes <nemes.bence1@gmail.com>
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
This is useful for the cases when original urls no longer exist, but nixpkgs still references them.
Change-Id: Ibfbb6784876efe4fc8f65fac84ff9dcb72e78ff7
Reviewed-on: https://cl.snix.dev/c/snix/+/30570
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
Autosubmit: Vova Kryachko <v.kryachko@gmail.com>
We don't want to crash the evaluator when an invalid regex is passed,
but instead display a user-facing error message. This CL does that.
Change-Id: I81fd8e342fc877344f8d2a3704ef53caf5190aa3
Reviewed-on: https://cl.snix.dev/c/snix/+/30588
Reviewed-by: Vova Kryachko <v.kryachko@gmail.com>
Tested-by: besadii
Reviewed-by: Bence Nemes <nemes.bence1@gmail.com>
Some analysing what is going on in nixpkgs revealed that we recompiled the same
regexes many times, and there aren't that many different regexes anyways.
This was actually visible in flamegraphs, now it's not anymore.
Fixes https://git.snix.dev/snix/snix/issues/151
Change-Id: Ia04b1833fec083017aebac99cdae7e91148966c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13464
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
(cherry picked from commit aa1eca36c3b3c18d96ba3081d7053b4c639e2f17)
Reviewed-on: https://cl.snix.dev/c/snix/+/30587
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Bence Nemes <nemes.bence1@gmail.com>
All these functions do blocking IO, and don't yield back to the
executor, so we cannot invoke them directly, but have to use
spawn_blocking and a channel.
Instead of plainly reverting cl/30575, this keeps potential errors being
sent as the last element of the stream.
We need to make our error construction a bit more ergonomic, potentially
allow them to wrap other errors instead of the madness this currently
is, but this is something for a later CL.
Change-Id: Ifb05871741813a389ac00b4f2c468f984a689a18
Reviewed-on: https://cl.snix.dev/c/snix/+/30586
Reviewed-by: Vova Kryachko <v.kryachko@gmail.com>
Tested-by: besadii
fuse-backend-rs spawns multiple threads, but these don't automatically
inherit the tokio runtime context.
Spawning tasks from "the wrong thread" would then cause a panic, as seen
in https://git.snix.dev/snix/snix/issues/147.
However, cl/30575 didn't really fix this, it only removed one place
spawning tasks, without fixing the underlying issue.
PathInfoService / BlobService / DirectoryService are expected to spawn
tasks. We need to simply invoke `.enter()` from all worker threads, and
that's what this CL does.
An alternative would be to manually (re)-enter inside every function of
the FileSystem trait, but that'd be very messy.
A similar fix needs to end up in the virtiofs implementation, but we
don't have control over the (single) thread being spawned by
VhostUserDaemon there, so cannot just enter the runtime context there,
so virtiofs will stay broken for now.
Maybe it's time to re-architect this a bit - have our FileSystem impl
be little code and call to sync endpoints to do the actual work, which
is then handled by workers on another thread - but that's left for
another CL.
Change-Id: I58cdbd952f4ecc39bdc2f2fa69a788caa0cc78ba
Reviewed-on: https://cl.snix.dev/c/snix/+/30585
Tested-by: besadii
Reviewed-by: Vova Kryachko <v.kryachko@gmail.com>
This is snix-castore, not snix-store.
Change-Id: I4c77adbc8c0c7c450075231ae4ef03ddfa0ea81c
Reviewed-on: https://cl.snix.dev/c/snix/+/30584
Tested-by: besadii
Reviewed-by: Vova Kryachko <v.kryachko@gmail.com>
The types are different for `gerrit stream-events` and webhooks, so
switch to a fork of go-gerrit containing their definitions.
https://github.com/andygrunwald/go-gerrit/pull/189 is the upstream PR.
Change-Id: I24136af2f2cf5655f2a8278632a3b0f52aa6adcc
Reviewed-on: https://cl.snix.dev/c/snix/+/30544
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Changelog: https://www.gerritcodereview.com/3.12.html
We are skipping over the 3.11.1, 3.11.2, 3.11.3 minor releases which
remains available.
This bump was already tested on another Gerrit instance.
No manual intervention is required.
Change-Id: Ia3ce1f1cda36abe6da4edd4210260f664f7b3672
Signed-off-by: Raito Bezarius <raito@lix.systems>
Reviewed-on: https://cl.snix.dev/c/snix/+/30576
Autosubmit: Ryan Lahfa <ryan@lahfa.xyz>
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
Fixes a panic in `list` impl of PathInfoService for fuse/virtiofs:
```
thread 'vring_worker' panicked at store/src/pathinfoservice/redb.rs:136:9:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
```
Closes#147.
Change-Id: I2bed5157b30fa185276dcaefd35277159a01fe2d
Reviewed-on: https://cl.snix.dev/c/snix/+/30575
Reviewed-by: Florian Klink <flokli@flokli.de>
Tested-by: besadii
the subgid maps are not mapped per group, but per user
Change-Id: Iaf367cff159109eefb5ef99d58033082e6151c5d
Reviewed-on: https://cl.snix.dev/c/snix/+/30568
Reviewed-by: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Vova Kryachko <v.kryachko@gmail.com>
All outputs constructed by derivation_to_build_request use inputs_dir as
a prefix (so we cannot use StorePath::from_bytes, which only takes
the basename), and they are relative to their root, so we cannot use
StorePath::from_absolute_path either.
Construct the store paths by stripping inputs_dir early (right after
the call to derivation_to_build_request), and use them later.
Change-Id: I3874e11cf21159c05b02314d9baf26cc98ea7956
Reviewed-on: https://cl.snix.dev/c/snix/+/30569
Reviewed-by: Yureka <snix@yuka.dev>
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
This change adds basic scaffolding to allow configuring hashed_mirrors that will be used
by fetchurl to download artifacts by their hash, this is useful in case certain URLs are
no longer available but required to bootstrap nixpkgs stdenv.
These urls will have higher priority than the url specified in fetchurl(and friends) and
will be attempted before falling back to the actual url specified in fetchurl.
Change-Id: I589bdef609075f274cbdf6b26af602cafaa7496a
Reviewed-on: https://cl.snix.dev/c/snix/+/30567
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
This is only used in to_nix_nixbase32_string, and in one place in
narinfo2parquet (though it might be sliced wrongly there too).
This is a partial revert of cl/12041, at least for the narinfo2parquet
parts.
Change-Id: Ic8c57373f7ab79043a491267e8484fa8399cea04
Reviewed-on: https://cl.snix.dev/c/snix/+/30564
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
We use this for both encoded and unencoded strings (and the error
message was missing), so this usize is pointless.
Change-Id: Id2a1000f1b232896605cdd34349ee114a67dc268
Reviewed-on: https://cl.snix.dev/c/snix/+/30563
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
This one is overkill. `NixHashResult<NixHash>` takes exactly as many
characters as `Result<NixHash, Error>`, so removing the type alias
actually removes the total amount of code.
The only external reference to it is somewhere that should probably live
in nixhash::ca_hash.
Change-Id: I0c4a149294d33129a67cb1b699cc8a645c7c18e1
Reviewed-on: https://cl.snix.dev/c/snix/+/30562
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: edef <edef@edef.eu>
The test code was way too complicated. We had testcases manually
constructing different NixHash as an input, extracted digest and algo,
then manually encoded them with various encodings, to then compare to
itself.
Instead, write out these different string inputs as explicit testcases.
Change-Id: I2adeedcb9ddc8b3d50f8bdab09a1e95198cda402
Reviewed-on: https://cl.snix.dev/c/snix/+/30560
Reviewed-by: edef <edef@edef.eu>
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
We can reject early if invalid keys are provided in the attrset, no need
to look at values already.
Also, restructure the code parsing and extracting a sha256 by
destructuring the enum, rather than grabbing a slice and trying to
convert to a fixed-size array.
Change-Id: I1bb067133e398626df25b9c1cf99926c6d836a19
Reviewed-on: https://cl.snix.dev/c/snix/+/30559
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: besadii
It was a bit confusing to construct NixHash, having them as separate
functions in the module itself, rather than in the NixHash impl.
Also the names were very inconsistent.
This renames parsers to `from_$format_$encoding` and format methods to
`to_$format_$encoding`. It also adds / moves around a few docstrings,
explaining the formats and encodings in the struct docstring itself.
from_str is changed to accept Option<HashAlgo>, not Option<&str>, and
the otherwise unused `from_nix_hash_string` is folded into from_str.
We also simply use from_sri in from_str, as the error path there doesn't
allocate anymore.
Similarly, the from_nix_str function was only a helper function used to
parse a subset of the formats supported in the NixHash::from_str method.
We shouldn't be using it outside of there, all usages (only in tests)
have been replaced with NixHash::from_algo_and_digest.
Change-Id: I36128839dbef19c58b55d5dc5817e38e37a483cc
Reviewed-on: https://cl.snix.dev/c/snix/+/30554
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: edef <edef@edef.eu>
Tested-by: besadii
Especially the various specific format serializers/deserializers with
used in path_info.rs shouldn't be living there, but in NixHash, so they
can be used by other consumers of the library wanting to restrict to a
certain format.
Change-Id: Id43ba96e3f6ec68999f028854b625d5335d71554
Reviewed-on: https://cl.snix.dev/c/snix/+/30556
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
This is only used inside NixHash::to_nix_hex_string().
Change-Id: I7c9c0cd7d4feaa41b0861bb5c0e99a47ec0caac1
Reviewed-on: https://cl.snix.dev/c/snix/+/30555
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
This looks more readable like this.
Change-Id: Iaa750fae66c7263612f169405eb7d38fb9541b04
Reviewed-on: https://cl.snix.dev/c/snix/+/30552
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: besadii
This error is used for invalid digest lengths for a passed HashAlgo, not
just when they're encoded (as can be seen in from_algo_and_digest).
Change-Id: I7604846ae133df1be516a1f7ab28efd2a5775145
Reviewed-on: https://cl.snix.dev/c/snix/+/30551
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
This is not used anywhere, and a bit surprising. Consumers can just use
from_algo_and_digest.
Change-Id: Id4fca98568b1967899fb7428e6767aa993e70c96
Reviewed-on: https://cl.snix.dev/c/snix/+/30550
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: besadii
This was decoding nixbase32, not hex. Its only consumer (in ca_hash.rs)
was right in its docstring about how it behaves, only was calling the
wrongly-named function.
Change-Id: I97ea273706ba818d16a61b1574989db800f78ead
Reviewed-on: https://cl.snix.dev/c/snix/+/30553
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
This changes RootNodes::list to return a BoxStream<'static, _>, and then
drops all the mpsc sender / receiver complexity we were having.
There's also no need to worry about channel buffer sizes - all current
RootNodes implementations are immediately ready to yield new elements in
the stream. Assuming there's new implementations that do take some time,
we can deal with buffer sizes on the producer size, which might know its
own batch sizes better.
RootNodes now doesn't need to implement Clone/Send anymore, and can have
non-static lifetimes. As long as its the list method returns a
BoxStream<'static>, we're fine with all that.
On a first look, this seems like we now need to do more cloning upfront
for the BTreeMap and Directory RootNodes impls. However, we already
had to clone the entire thing at `self.root_nodes_provider.clone()`, and
then did it again for each element.
Now we get an owned version of the data whenever a list() call happens,
and then just move owned things around.
Change-Id: I85fbca0e1171014ae85eeb03b3d58e6176ef4e2d
Reviewed-on: https://cl.snix.dev/c/snix/+/30549
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: besadii
There's multiple places where we peek at the node to construct a
FileType, so move this into a helper.
Also, get rid of a async move which didn't move, and use .ok_or_else to
make things a bit more readable.
Change-Id: I2d24a3291029fdc12e0049398d8d51111e22d3cf
Reviewed-on: https://cl.snix.dev/c/snix/+/30548
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: besadii
We can now use async closures for this.
Change-Id: Iccbe86998726be139e81749745c37eb9f475693c
Reviewed-on: https://cl.snix.dev/c/snix/+/30546
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Even without nix/store in here, all output paths need to be write-able.
Change-Id: Ibeeba503844dee78de11fd2aa79b3ad207795059
Reviewed-on: https://cl.snix.dev/c/snix/+/30542
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Vova Kryachko <v.kryachko@gmail.com>
Nix's `builtin.placeholder` function produces output paths that are not
known ahead of time, so before propagating these values into the build
we need to replace them in all env variables and arguments to their
corresponding output store paths.
fix#101
Change-Id: I2670c749f2c578e276d698e511598a76a99ebb96
Reviewed-on: https://cl.snix.dev/c/snix/+/30310
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
Autosubmit: Vova Kryachko <v.kryachko@gmail.com>
And allow gerrit01 to send these hooks over to irccat running on meta01.
Issue: https://git.snix.dev/snix/snix/issues/74
Change-Id: Ic5835734b32e8e5a46225e68d4124d55c002d663
Reviewed-on: https://cl.snix.dev/c/snix/+/30527
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Ryan Lahfa <ryan@lahfa.xyz>
This is a listener for gerrit events, sent by their "webhooks" plugin,
as well as a NixOS module to deploy it.
Issue: https://git.snix.dev/snix/snix/issues/74
Change-Id: I65c5c5a991e6b1f4f330b3439c8a25aec3f1b484
Reviewed-on: https://cl.snix.dev/c/snix/+/30526
Reviewed-by: Ryan Lahfa <ryan@lahfa.xyz>
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
cl/30249 changed this to Postmark, and it was unconfigured before.
Change-Id: I89eb49dbb8a3cb81135ae01c98379151e32ecd7c
Reviewed-on: https://cl.snix.dev/c/snix/+/30528
Tested-by: besadii
Reviewed-by: Ryan Lahfa <ryan@lahfa.xyz>
Autosubmit: Florian Klink <flokli@flokli.de>
