This one is overkill. `NixHashResult<NixHash>` takes exactly as many
characters as `Result<NixHash, Error>`, so removing the type alias
actually removes the total amount of code.
The only external reference to it is somewhere that should probably live
in nixhash::ca_hash.
Change-Id: I0c4a149294d33129a67cb1b699cc8a645c7c18e1
Reviewed-on: https://cl.snix.dev/c/snix/+/30562
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: edef <edef@edef.eu>
The test code was way too complicated. We had testcases manually
constructing different NixHash as an input, extracted digest and algo,
then manually encoded them with various encodings, to then compare to
itself.
Instead, write out these different string inputs as explicit testcases.
Change-Id: I2adeedcb9ddc8b3d50f8bdab09a1e95198cda402
Reviewed-on: https://cl.snix.dev/c/snix/+/30560
Reviewed-by: edef <edef@edef.eu>
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
We can reject early if invalid keys are provided in the attrset, no need
to look at values already.
Also, restructure the code parsing and extracting a sha256 by
destructuring the enum, rather than grabbing a slice and trying to
convert to a fixed-size array.
Change-Id: I1bb067133e398626df25b9c1cf99926c6d836a19
Reviewed-on: https://cl.snix.dev/c/snix/+/30559
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: besadii
It was a bit confusing to construct NixHash, having them as separate
functions in the module itself, rather than in the NixHash impl.
Also the names were very inconsistent.
This renames parsers to `from_$format_$encoding` and format methods to
`to_$format_$encoding`. It also adds / moves around a few docstrings,
explaining the formats and encodings in the struct docstring itself.
from_str is changed to accept Option<HashAlgo>, not Option<&str>, and
the otherwise unused `from_nix_hash_string` is folded into from_str.
We also simply use from_sri in from_str, as the error path there doesn't
allocate anymore.
Similarly, the from_nix_str function was only a helper function used to
parse a subset of the formats supported in the NixHash::from_str method.
We shouldn't be using it outside of there, all usages (only in tests)
have been replaced with NixHash::from_algo_and_digest.
Change-Id: I36128839dbef19c58b55d5dc5817e38e37a483cc
Reviewed-on: https://cl.snix.dev/c/snix/+/30554
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: edef <edef@edef.eu>
Tested-by: besadii
Especially the various specific format serializers/deserializers with
used in path_info.rs shouldn't be living there, but in NixHash, so they
can be used by other consumers of the library wanting to restrict to a
certain format.
Change-Id: Id43ba96e3f6ec68999f028854b625d5335d71554
Reviewed-on: https://cl.snix.dev/c/snix/+/30556
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
This is only used inside NixHash::to_nix_hex_string().
Change-Id: I7c9c0cd7d4feaa41b0861bb5c0e99a47ec0caac1
Reviewed-on: https://cl.snix.dev/c/snix/+/30555
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
This looks more readable like this.
Change-Id: Iaa750fae66c7263612f169405eb7d38fb9541b04
Reviewed-on: https://cl.snix.dev/c/snix/+/30552
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: besadii
This error is used for invalid digest lengths for a passed HashAlgo, not
just when they're encoded (as can be seen in from_algo_and_digest).
Change-Id: I7604846ae133df1be516a1f7ab28efd2a5775145
Reviewed-on: https://cl.snix.dev/c/snix/+/30551
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
This is not used anywhere, and a bit surprising. Consumers can just use
from_algo_and_digest.
Change-Id: Id4fca98568b1967899fb7428e6767aa993e70c96
Reviewed-on: https://cl.snix.dev/c/snix/+/30550
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: besadii
This was decoding nixbase32, not hex. Its only consumer (in ca_hash.rs)
was right in its docstring about how it behaves, only was calling the
wrongly-named function.
Change-Id: I97ea273706ba818d16a61b1574989db800f78ead
Reviewed-on: https://cl.snix.dev/c/snix/+/30553
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
This changes RootNodes::list to return a BoxStream<'static, _>, and then
drops all the mpsc sender / receiver complexity we were having.
There's also no need to worry about channel buffer sizes - all current
RootNodes implementations are immediately ready to yield new elements in
the stream. Assuming there's new implementations that do take some time,
we can deal with buffer sizes on the producer size, which might know its
own batch sizes better.
RootNodes now doesn't need to implement Clone/Send anymore, and can have
non-static lifetimes. As long as its the list method returns a
BoxStream<'static>, we're fine with all that.
On a first look, this seems like we now need to do more cloning upfront
for the BTreeMap and Directory RootNodes impls. However, we already
had to clone the entire thing at `self.root_nodes_provider.clone()`, and
then did it again for each element.
Now we get an owned version of the data whenever a list() call happens,
and then just move owned things around.
Change-Id: I85fbca0e1171014ae85eeb03b3d58e6176ef4e2d
Reviewed-on: https://cl.snix.dev/c/snix/+/30549
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: besadii
There's multiple places where we peek at the node to construct a
FileType, so move this into a helper.
Also, get rid of a async move which didn't move, and use .ok_or_else to
make things a bit more readable.
Change-Id: I2d24a3291029fdc12e0049398d8d51111e22d3cf
Reviewed-on: https://cl.snix.dev/c/snix/+/30548
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: besadii
We can now use async closures for this.
Change-Id: Iccbe86998726be139e81749745c37eb9f475693c
Reviewed-on: https://cl.snix.dev/c/snix/+/30546
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Even without nix/store in here, all output paths need to be write-able.
Change-Id: Ibeeba503844dee78de11fd2aa79b3ad207795059
Reviewed-on: https://cl.snix.dev/c/snix/+/30542
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Vova Kryachko <v.kryachko@gmail.com>
Nix's `builtin.placeholder` function produces output paths that are not
known ahead of time, so before propagating these values into the build
we need to replace them in all env variables and arguments to their
corresponding output store paths.
fix#101
Change-Id: I2670c749f2c578e276d698e511598a76a99ebb96
Reviewed-on: https://cl.snix.dev/c/snix/+/30310
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
Autosubmit: Vova Kryachko <v.kryachko@gmail.com>
This change makes BUILD_TOP to point to /build, which is what nix does.
Change-Id: I4ffef67aff0665d13859378a86329291a53d4ea0
Reviewed-on: https://cl.snix.dev/c/snix/+/30500
Reviewed-by: Florian Klink <flokli@flokli.de>
Autosubmit: Vova Kryachko <v.kryachko@gmail.com>
Tested-by: besadii
Rather than having separate branches, just make it part of the state
machine discipline.
Change-Id: Ib21456227515506495ca06ac2a8a529d04f95fde
Reviewed-on: https://cl.snix.dev/c/snix/+/30496
Reviewed-by: Brian Olsen <brian@maven-group.org>
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
We should never return `Poll::Pending` without having received it from
the underlying reader.
Change-Id: I8c79c0243dc45889c1df478712971ef930e5f3a9
Reviewed-on: https://cl.snix.dev/c/snix/+/30498
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
Legitimate zero-length reads could cause spurious unexpected EOF,
since we implicitly assumed buffers always have remaining capacity.
For the buffered case, `consume(0)` could cause panics after either
`poll_fill_buf` or `poll_read` had returned `Poll::Pending`.
The bytes_read/with_limited logic receives a stylistic cleanup to make
it obvious that bytes_read is always written before being used.
Change-Id: I46aa47113309552dcef9532b5d4009d2186db9cd
Reviewed-on: https://cl.snix.dev/c/snix/+/30492
Tested-by: besadii
Reviewed-by: Brian Olsen <brian@maven-group.org>
Reviewed-by: Florian Klink <flokli@flokli.de>
This prevents framing confusion, which would otherwise lead to a
trivial confused deputy attack. See issue #120.
The NixFramedReader state machine has been refactored to simplify
its internal logic and accurately account for EOF conditions.
End-of-stream is fused, and unexpected EOF on the underlying reader
is returned as UnexpectedEof, though we don't fuse those ourselves.
We also ensure that the underlying reader does not swap the ReadBuf;
this would otherwise supply a primitive for converting uninitialised
mutable memory into `&mut [u8]` without initialisation, thus allowing
undefined behaviour to be triggered from safe code.
Change-Id: I05ddb7e3ca57b3363f56c0d9b43d5a641748ca36
Reviewed-on: https://cl.snix.dev/c/snix/+/30380
Reviewed-by: Brian Olsen <brian@maven-group.org>
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
This is mostly a WIP commit, to demonstrate bugs properly. See #120.
The tests are marked `#[should_panic]`, since they are intended to fail.
Change-Id: I39f1d66742e6629ccb889da8ef1199117b91b126
Reviewed-on: https://cl.snix.dev/c/snix/+/30490
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
The messages we can throw are not necessarily UTF-8 strings. The
to_string() in there did store the result of the Display impl, which is
a quoted string.
Change-Id: I65a77ccc7f2d62ff06a2a9458cdb7e7292f132b0
Reviewed-on: https://cl.snix.dev/c/snix/+/30489
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Bence Nemes <nemes.bence1@gmail.com>
This tests deep forcing happens in lexicographic key order, by comparing
the returned error from the evaluator. It's not possible to observe this
from inside nixlang, which is why we use one_offs.rs here.
Change-Id: I73085addca3a4df20bc23f9fced458758af5b391
Reviewed-on: https://cl.snix.dev/c/snix/+/30488
Reviewed-by: Bence Nemes <nemes.bence1@gmail.com>
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
Using hashmap seems to give a decent speedup overall.
hello outpath time: [528.01 ms 529.17 ms 530.64 ms]
change: [-22.932% -22.563% -22.181%] (p = 0.00 < 0.05)
Performance has improved.
firefox outpath time: [4.7647 s 4.8149 s 4.8917 s]
change: [-21.251% -20.408% -18.914%] (p = 0.00 < 0.05)
Performance has improved.
But it slows down derivation parsing by about 1-1.5%
Added another attr merge benchmark that helped me while profiling,
not sure if we want to keep that.
Change-Id: Icb9f1e2d40bbb7150af1b8df192bf3c860bae79b
Reviewed-on: https://cl.snix.dev/c/snix/+/30309
Tested-by: besadii
Reviewed-by: Florian Klink <flokli@flokli.de>
moved to #138 and #139.
Change-Id: I3ad3dc5ab0c38ba4ed0ac43d5c492f802be61ed8
Reviewed-on: https://cl.snix.dev/c/snix/+/30481
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Combinators are tracked in #135, a followup for `CombinedBlobService` is
tracked in #136.
User-facing composition config is tracked in #137.
Everything else mostly already landed with the rest of the store
composition, so can be dropped.
Change-Id: I3e0aee409f8314b1a0582541fd5f1b8b50405ce5
Reviewed-on: https://cl.snix.dev/c/snix/+/30480
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
A mention of these different builders is included in the a footnote
in the documentation, and various issues for the different TODOs were
created:
- #128 Implement bwrap-based Builder
- #129 Implement gVisor-based builder
- #130 Implement Cloud Hypervisor-based builder
- #131 OCI builder: add preflight checks
- #132 BuildService: refactor to be more granular
Change-Id: I349b799e233ba8bef39a139cf2453d3214bb69b3
Reviewed-on: https://cl.snix.dev/c/snix/+/30474
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
This was most likely meant to refer to `exportReferencesGraph`, not
`fetchClosure`. `fetchClosure` is not used in nixpkgs - I created #127
still.
Issue #44 is extended to mention `ExportedPathInfo`.
Change-Id: Id898cb381db02c83888dc395cf3ab01ae6baf2aa
Reviewed-on: https://cl.snix.dev/c/snix/+/30473
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
This is very generic and not helpful.
Change-Id: Ie851e0e293023ab1794c6815e0a0e188471f509b
Reviewed-on: https://cl.snix.dev/c/snix/+/30471
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
This was already migrated to use a BigtableParameters struct, similar to
other backends.
Change-Id: Icc8a4902a6f24ce4a7f965abc800726b09030cb3
Reviewed-on: https://cl.snix.dev/c/snix/+/30470
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
This was migrated to #122 and #123.
Change-Id: I5196a12530fe420c7682312774e14807df688928
Reviewed-on: https://cl.snix.dev/c/snix/+/30467
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
These don't make sense without a parent, move these one level up.
Change-Id: I492e43da1d1a429e7c46b65b0c676d5d8c54fdf6
Reviewed-on: https://cl.snix.dev/c/snix/+/30466
Tested-by: besadii
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Autosubmit: Florian Klink <flokli@flokli.de>
This has been migrated to #64 a while ago.
Change-Id: Iec15043650284ac7c2cb62863028f360675bdc82
Reviewed-on: https://cl.snix.dev/c/snix/+/30465
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
This is slowly being plucked apart and migrated to more suitable places.
Change-Id: Ib4f4e76601a657cfef85dc759f8ec9bde4eadb86
Reviewed-on: https://cl.snix.dev/c/snix/+/30464
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Back when initially working on this, having all info about the Build in
one struct seemed a good idea for some future CI interface, but right
now this simply raises more questions and is quite theoretic.
Let's drop it for now, we can reintroduce it, or other request methods
when we get to it.
Change-Id: I105a8d5ae8bd7e0d5f8ee3e7edf2597100b43119
Reviewed-on: https://cl.snix.dev/c/snix/+/30425
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Vova Kryachko <v.kryachko@gmail.com>
Put it in Guides, as it provides some examples at the end as well.
Change-Id: Ic5cd78bcda09c3bb82eeaa88ff0c959c4c876bd7
Reviewed-on: https://cl.snix.dev/c/snix/+/30445
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Yureka <snix@yuka.dev>
Tested-by: besadii
Part of #114
cargo fix wanted to rewrite `if let else` to match statements, but i
reverted them as they dont belong in this cl.
There weren't any warnings about locks (relative drop order changed in
2024)
Change-Id: I9c851ef8e214a481cbe7b4cf9b2634b5d56970d4
Reviewed-on: https://cl.snix.dev/c/snix/+/30369
Autosubmit: Bence Nemes <nemes.bence1@gmail.com>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Florian Klink <flokli@flokli.de>
Tested-by: besadii
This allows piping NAR data through a reader, and writing it back out to
a writer.
It can be used to validate a NAR to be syntactically correct, or to read
exactly to the end of a NAR file if the size is not given externally.
Change-Id: I0fc8d58e68783400d1cfee75c860138915974f3d
Reviewed-on: https://cl.snix.dev/c/snix/+/30423
Tested-by: besadii
Reviewed-by: edef <edef@edef.eu>
Autosubmit: Florian Klink <flokli@flokli.de>
This changes the BuildService trait to return a typed `BuildResult`,
which bundles the refscan info alongside the castore nodes.
The proto type is renamed to BuildResponse, to better map to gRPC
semantics.
In proto land, we don't send the name for outputs anymore, be it the
full path or the last component, as there's never been a guarantee this
is a valid PathComponent. That entry is now required to be anonymous.
The path of an output can be retrieved by looking at the original
BuildRequest.
Change-Id: If5ce3a009cd3dd6bb6505cd51d5f4deda261ea85
Reviewed-on: https://cl.snix.dev/c/snix/+/30387
Autosubmit: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: Vova Kryachko <v.kryachko@gmail.com>
