This change implements the serialization part that is needed to
implement the nix daemon protocol. Previously was add deserialization
and derivers for that and this then adds the other part of that equation
so that you can write types that can then be read using deserialization.
Change-Id: I2917de634980a93822a4f5a8ad38897b9ce16d89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12729
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is now supported in the standard library via std::sync::LazyLock, but
requires some manual shuffling around of code.
I found at least one dead variable along the way, which I deleted.
Change-Id: I8600c87c49078fb5ff72671994c77b919259e67b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12608
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
We always read the length before returning a BytesReader, so len()
cannot be called before the length is read.
Change-Id: Ifa1cbc2c0923dd24d59e0af5e135ab8ed7314aa9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12453
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
When only the wire feature was enabled two methods were unused and so
would faild `cargo check --no-default-features --features wire`.
This feature flags those two methods on async feature since that is the
only place they are used.
Change-Id: I6ec18a670e3c6e3ecee8d1417c99f1a5084e0ae7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12346
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Add a trait for deserializing a type from a daemon worker connection.
This adds the NixDeserialize trait which is kind of like the serde
Deserialize trait in that individual types are meant to implement it
and it can potentially be derived in the future.
The NixDeserialize trait takes something that implements NixRead as
input so that you can among other things mock the reader.
Change-Id: Ibb59e3562dfc822652f7d18039f00a1c0d422997
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11990
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We were wrongly using `buf.initialized()` which contains more than the
filled portion to compute the number of bytes read during a poll call.
This made us go into the trailer reading state too early and finally
failing due to invalid trailer data.
Fixes b/405.
Co-authored-by: Florian Klink <flokli@flokli.de>
Change-Id: I66ba6e2116389e6b97305f85c4e0922195000e45
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11978
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
See https://github.com/NixOS/nixpkgs/pull/323753 for details.
Changes:
* git: temporarily comment out dottime patch (it doesn't apply, but it's not critical)
* third-party/cgit: use an older git version where dottime patch still applies
* 3p/crate2nix: remove crate2nix patches included in latest release
* tvix: remove unneeded defaultCrateOverrides (upstreamed to nixpkgs)
* tvix: regenerate Cargo.nix
* tvix/nix-compat: remove unnused AtermWriteable::aterm_bytes pub(crate) function
* tvix/nix-compat: remove redundant trait bounds
* tvix/glue: use clone_into() to set drv.{builder,system}
* tools/crate2nix: apply workaround for https://github.com/numtide/treefmt/issues/327
* toold/depotfmt: expose treefmt config as passthru
* tools/crate2nix: undo some more hacks in the crate2nix-check drv
Change-Id: Ifbcedeb3e8f81b2f6ec1dbf10189bfa6dfd9c75c
Co-Authored-By: Florian Klink <flokli@flokli.de>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11907
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Makes the following build:
`cargo test --no-default-features --features async`
`cargo test --no-default-features --features wire`
Change-Id: I47ba0c944f08895f67ed3b861706ef2e4ba384b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11739
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We reuse the prev_name allocation for Entry, instead of allocating and
returning a separate Vec.
We encode the `prev_name: None` case as an empty vector, since we don't
allow empty names anyway, and the sorting is equivalent.
Change-Id: I975b37ff873805f5ff099bc82128706891052247
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11607
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
For small bytestrings (like NAR names), we can read into a preallocated
fixed-size buffer, instead of allocating a Vec every time.
Change-Id: Id8da9e9cea99c814361230c0ec02606b731c79a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11606
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
If our underlying reader supports AsyncBufRead, then we can too.
Change-Id: If4b948c983400ca591c1c475bbcf7dc00d562040
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11545
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Even if we have an aligned (or zero) size (and thus have no padding),
we may still have a non-padding trailer.
Since an exact read of only the user data bytes must always read the
trailer bytes as well, this has to happen for aligned sizes too.
For zero-sized trailers, we solve this by reading immediately, since no
read is required of the user at all.
user_len thus becomes a NonZeroU64, and the definition of "body data"
is adjusted to always exclude the final block.
Change-Id: I2c6bb51270fb22ad72a65076a3468673ab56ef68
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11597
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We're using this in the NAR reader now.
Change-Id: I28f17b1ccedd62ffcaf2fa32b517f16bcd036d94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11603
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We specifically structured the code this way to allow using
`this.filled` as-is, so we should use it.
Change-Id: I7e11bddceb4d5f37b1dd4c453a9d53b85fc1f6c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11602
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is a first cut at the async NAR reader, with some rough edges.
Poisoning is left unimplemented for now, pending future work.
Change-Id: Ifaafe0581a5e0e165a13357b909fb441f7bd8bab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11524
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This allows using BytesReader with a custom tag, eg the closing parens
for the NAR reader.
No public constructor is provided for custom-tagged readers, since this
feature isn't public API.
Change-Id: I82e73d064edc4b6783ead1d6fe46a5b35f45c844
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11543
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
u64 is an inappropriate type for something memory-sized, and most
callers end up with off-by-ones when using `..` rather than `..=`,
including the tests for the module itself.
Change-Id: If3b7bea27eb0a6c01e0a5d7e64966acbbb664268
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11550
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
These may as well be inlined, and hardly need tests, since they just
alias AsyncReadExt::read_u64_le / AsyncWriteExt::write_u64_le.
Boolean reading is worth making explicit, since callers may differ on
how they want to handle values other than 0 and 1.
Boolean writing simplifies to `.write_u64_le(x as u64)`, which is also
fine to inline.
Change-Id: Ief9722fe886688693feb924ff0306b5bc68dd7a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11549
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
with_size only existed to allow separating the phases of reading size
and reading data, and similarly the Size state only existed to allow
folding size reading into poll_read.
Neither of these are necessary if we make the constructor async,
and handle the size reading there.
As a bonus, this makes BytesReader::len sensibly typed, and permits
implementing is_empty, as Clippy demands of us.
Change-Id: I72173ec06d60b7998d16a3ecfc5e6ac5424bbed3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11542
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We don't actually hold a Tag, we just want to bind the type.
Change-Id: Ida67c026f852ed54c3f18df914cf5c31e6227fd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11541
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The (min, max) pair is already a RangeInclusive in essence, so we might
as well represent it that way.
Change-Id: I2f67f3c47dc36b87e866ff5dc2e0cd28f01fbb04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11540
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The TrailerReader has no purpose separate from BytesReader, and the
code gets a fair bit simpler this way.
EOF handling is simplified, since we just rely on the implicit
behaviour of the existing case.
Change-Id: Id9b9f022c7c89fbc47968a96032fc43553af8290
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11539
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The API is a bit odd here, because we don't have a distinct type for a
known-length reader.
Change-Id: I4a1dd07fbed0a400004dbe4aa2095c51898ad3bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11538
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
We separate ingesting the trailer block into a Future of its own,
parametrised on the specifics of the trailer pattern.
This is intended to be used for future work on an async NAR reader,
which needs to read a terminating parenthesis as well as the regular
padding.
Thanks to @griff for suggesting separating the ingestion into its own
Future.
Change-Id: I36c2503baa67937046a63e9bf0cfc38201394025
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11522
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Pin<&mut T> is DerefMut when T: Unpin, so we don't actually need to
explicitly call get_mut.
Change-Id: Iaa312ec49c87100010e09c94f319e57e31da0cd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11520
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We don't use it in the reader anymore.
Change-Id: I98fe204a747711464e9e7ca17df06fa9854eb344
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11519
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We now *never* return the final bytes until we've read the padding
in full, so read_exact is safe to use.
This is implemented by TrailerReader, which splits the phases of
reading (and validating) the final 8-byte block, and providing
the contained payload bytes to the caller.
Change-Id: I0d05946a8af9c260a18d71d2b763ba7a68b3c27f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11518
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We already require R: Unpin in the constructor, so there's not much use
to pin projection.
Change-Id: Ia7bf734dc3aa86ffa6d1d5de778939baa9676bb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11516
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is semantically a RangeInclusive, since we can only have
0..=u64::MAX at most, and monomorphising on the bounds doesn't
buy us anything.
Change-Id: Ib601d7fd77d703d6c8c5ec27ac9e67bb122ce1c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11515
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is a ReadBuf, not a BufRead.
Change-Id: Ie80e894f4b24b77cdd60409ddfaa66dae0ffeec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11511
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
While the compiler might realize it's a good idea to inline these,
especially considering this is in the same crate [1], it doesn't hurt to
add that annotation too.
Suggested in cl/11385.
[1]: https://nnethercote.github.io/perf-book/inlining.html
Change-Id: If679fce0e1aab9ec681ba60f689d386d6fd92d61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11403
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Move everything bytes-related into its own module, and re-export
both bytes and primitive in a flat space from wire/mod.rs.
Expose this if a `wire` feature flag is set. We only have `async` stuff
in here.
Change-Id: Ia4ce4791f13a5759901cc9d6ce6bd6bbcca587c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11389
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
