Commit graph

17 commits

Author SHA1 Message Date
Florian Klink
36e4d017f5 chore(snix): s/tvix/snix/
Change-Id: Iae961416eea0a38bc57df7b736f6dda5903b0828
2025-03-17 17:15:08 +00:00
Florian Klink
23a7ead71a feat(nix-compat): allow parsing and emitting log lines
This adds code allowing to parse (and emit) the "internal-json" log
format that Nixcpp emits.

Move the Verbosity struct from worker_protocol.rs out into
src/log/mod.rs, it's not only used there.

Change-Id: Ia35e739c83e06080a677b576616bdd0d4a821cf4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13167
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: edef <edef@edef.eu>
2025-03-04 08:17:05 +00:00
Florian Klink
a512f16424 chore(nix-compat): bump to nom 8.x
See 72dd5818b7/CHANGELOG.md
for the nom changelog.

Most notably, there's now a .parse() to be added:

`combinator(arg)(input)` -> `combinator(arg).parse(input)`

There also doesn't need to be a tuple combinator (it's implemented on
tuples directly).

This also refactors the string / byte field parsing parts, to make them
more concise.

Change-Id: I9e8a3cedd07d6705be391898eb6a486fb8164069
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13193
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: Brian Olsen <me@griff.name>
2025-03-04 08:17:05 +00:00
Florian Klink
0fd8aa8b0d chore(users/edef): regenerate Cargo lock files
Change-Id: I379c9ad147e46ff218fed7256f52b960a105abe7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13191
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2025-03-02 18:47:47 +00:00
Florian Klink
661fe698a1 chore(tvix/tracing): bump to opentelemetry 0.28
According to https://github.com/open-telemetry/opentelemetry-rust/issues/1395#issuecomment-2612865575,
we can now remove the spawn_blocking calls and channel logic for
flushing / shutdown.

Change-Id: I0fabc1fe0968a736e7d44dfde41f4a6adac5e272
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13154
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2025-02-22 10:09:51 +00:00
Florian Klink
0715163779 chore(tvix): bump opentelemetry to 0.27
Change-Id: I3afbd8c94e0bc2cdf30da1886ae05b922f8a718f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12822
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-11-23 09:40:21 +00:00
Vova Kryachko
fa9c067dc9 feat(nix-daemon): Implement more nix daemon operations.
In particular QueryPathFromHashPart, QueryValidPaths, QueryValidDerivers

Change-Id: Ie6ad83cec5ce9580044b85e201e4e23394f87075
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12762
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
2024-11-13 21:21:57 +00:00
Vova Kryachko
b564ed9d43 feat(nix-daemon): Implement client handler.
This change includes only the basic nix handshake protocol handling and
sets up a client session. The only supported operation at this point is
SetOptions.

Additional operations will be implemented in subsequent cls.

Change-Id: I3eccd9e0ceb270c3865929543c702f1491768852
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12743
Autosubmit: Vladimir Kryachko <v.kryachko@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: Brian Olsen <me@griff.name>
2024-11-12 02:15:04 +00:00
edef
84a82f6f41 feat(users/edef/weave): use tracing_indicatif for progress
Progress bars :3

Change-Id: I770d0f8381521b6efc8b38c0db4d59c771887fee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12673
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-10-20 21:22:23 +00:00
edef
201d8f0cf2 feat(users/edef/weave): use FxHashSet and dedupe early
Deduping early saves a fair bit of memory, but the extra hashing is
costly.

We switch to FxHash, since we don't need a DoS-proof hash, but we do
need it to be *fast*.

Change-Id: Ic6b7010874c417862baa9b882593208c8dd1d5e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12648
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-10-19 17:01:38 +00:00
edef
06d2536eec feat(users/edef/weave): ingest roots in Parquet format
Parsing of store-paths.xz is now handled by //users/edef/fetchroots.

Change-Id: I78be5aada0c0a321ed79d80c9b615e5f997ac3e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12670
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-10-19 17:01:38 +00:00
edef
313899c291 refactor(users/edef/weave/swizzle): use polars streaming
This vastly reduces the memory requirements, so we can run in ~40G RAM.

Change-Id: I4952a780df294bd852a8b4682ba2fd59b9bae675
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12667
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-10-19 17:01:38 +00:00
edef
bdc2891053 refactor(users/edef/weave): simplify OwningRef handling
We switch to working with the underlying Arc<Bytes<u8>> type, since
Buffer<u8> is already essentially an OwningRef-esque subslice.

Because we're now working with an exposed Arc directly, we don't need
to have our own `unsafe impl StableAddress` any more.

Change-Id: I9ce2edc6899177145e15b72aa5380f708a62173c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12668
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-10-19 13:49:02 +00:00
edef
549c51f200 refactor(users/edef/weave): use swap rather than CAS
We only care about ordering on the same variable, and we rely on the
release barrier provided by rayon's thread joining.

The comparison failing is always an error path, and we're indifferent
about which thread it fails on.

Change-Id: I592a7eaae5b6935cf424c6576a49786f39909fb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12666
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-10-19 13:49:02 +00:00
edef
25671c284f refactor(users/edef/weave): simplify DoubleEndedIterator bound
Thanks, Clippy!

Change-Id: I116b478124b72e070e94150ee850532752f64d60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12669
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-10-19 13:49:02 +00:00
edef
dfff592784 fix(users/edef/weave): use safer_owning_ref
owning_ref has serious unsoundness.

Change-Id: Ie760697cd6399e6bc75f1ad17c9bb74adc077a35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12656
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
2024-10-17 22:47:41 +00:00
Vincent Ambo
bb34210a64 chore(tvix/tools): move weave to //users/edef
This is not a core Tvix tool, it's a tool that uses a Tvix component.

Change-Id: I705f2c4ab87f1512e005007c933e16b84ed4279f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12605
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2024-10-12 23:11:25 +00:00