- Update OWNERS in the root directory
- drop OWNERS in contrib/, snix/ and nix/ subdirectories
Change-Id: I5badfb950e3207e9a24cde7725bb4d9331f17ae4
Reviewed-on: https://cl.snix.dev/c/snix/+/30164
Tested-by: besadii
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Ryan Lahfa <masterancpp@gmail.com>
It's convenient for depending on sub-tree members of the current file as
well as gives access to siblings.
Change-Id: I74234cec6566177d88d3bc8507fa3f6ec789adb8
Reviewed-on: https://cl.snix.dev/c/snix/+/30098
Reviewed-by: adis bladis <adisbladis@gmail.com>
Tested-by: besadii
Tests were removed in cff6575948 to get rid of yants.
This reimplements a minimal runTestsuite feature set to support the readTree tests.
Fixes https://git.snix.dev/snix/snix/issues/71
Change-Id: I56e6e6e518cbbd6170b11f6d9ee8d62485dcb56e
Reviewed-on: https://cl.snix.dev/c/snix/+/30107
Reviewed-by: Florian Klink <flokli@flokli.de>
Tested-by: besadii
Reviewed-by: adis bladis <adisbladis@gmail.com>
Use the default emacs attribute over the versioned emacs28 attribute. On
current canon, this implies an upgrade to Emacs 29.
Emacs < 30 has some CVEs that have never been patched in those release
series. The next channel bump will upgrade pkgs.emacs to Emacs 30, so
just using the default alias seems to be the easiest solution.
See also <https://github.com/NixOS/nixpkgs/pull/386174>.
Change-Id: I1580bae138cc0801e0f3431b1a02c8a4585c2996
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13216
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Fixes execution of extraSteps on a machine that hasn't built the
pipeline and thus realised the extra step shell script. You can sort of
dump a script into `command` for buildkite as long as you escape any
variable access since buildkit would substitute those before entering
the step environment.
Change-Id: I8ddc0b80f6f568204ea6c80a118533bc11786473
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12992
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
readTree gives special treatment to the directory on which it is
invoked -- for example, it won't read *.nix files in that directory.
This commit adds the ability to disable this special treatment, which
remains the default behavior.
Example use case:
10029d3682
Change-Id: I306bea95f4d556f7090f3255e6da6bb410adbf57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12939
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
It's only used in my user directory and I don't feel it's generally
useful. There's so little to interpolate you may as well just check them
in.
Change-Id: I31dc9917c2b281c7d68388e1a32c8ef5179621df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12873
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
We allow strings with context that represent paths (since they qualify
as `pathLike`). While store path (names) may not contain any characters
that are meaningful in shell, they may contain directories and/or files
with such names since it's permissible in POSIX.
To fix this, we convert the given value `v` to a shell argument in two
stages:
1. Use `${v}` to coerce the value to a string while importing any
necessary paths to store.
2. Escape the resulting string for use as an argument.
Change-Id: Ib989b50df2a921c2abcd1ebc7ca0ff6e2bb79088
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12898
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
We don't need to calculate dependencies between and on targets that are
part of the parent target map since they will be skipped by buildkite
anyways. This speeds up 🦙 considerably for pipeline runs that have a
limited number of changed targets and a parent target map passed
in (i.e. pipeline runs of most CLs, but not canon runs). In my testing
it was about a minute faster (1/6 of the time 🦙 takes currently) for a
pipeline where under five drv targets changed. For the full
pipeline (i.e. no parentTargetMap) 🦙 takes about the same time as
before (it's a few seconds slower as is to be expected, but nothing
significant).
Change-Id: Ia5a80e142da8f40bc591e2c6cfaf48c325b2f577
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12818
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This bumps Gerrit to 3.10.0, and also introduces a new mechanism for
building it that should hopefully have some more stable hashes than the
previous bodgery.
In this world, we only cache what we explicitly want to. There are some
hooks implemented for `rules_java` and `rules_nodejs` (before version
6) that force use of local binaries; this means we can drop the use of
the FHSUserEnv and use the java and nodejs binaries provided by nixpkgs
instead.
detzip is deleted; it hasn't been used in yonks.
We also add https://gerrit-review.googlesource.com/c/gerrit/+/431977,
which bumps the SSHd version so that we can have U2F-based SSH keys.
Change-Id: Ie12a9a33bbb1e4bd96aa252580aca3b8bc4a1205
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11963
Reviewed-by: lukegb <lukegb@tvl.fyi>
Autosubmit: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
With custom phases, using `prompt` is not limited to the "release" phase
only, we only care about it not being in the "build" phase.
Change-Id: I8af27016865c728125dbe144d31006915b20b105
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11499
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
io/ioutil is deprecated, and the range expression can be simplified.
Change-Id: I73b85991faafa333bddf90adbdefe2006f8d409c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11460
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Want to get a few more signups than we allocated, in case somebody
drops out.
Change-Id: I832c4fdfa900f117092f5f7e8a5d8b7d256a94e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11450
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Most of the steps in our buildkite pipeline build derivations without
doing anything else. A lot of those derivations depend on each other.
Consequently, buildkite will schedule builds of derivations whose
dependencies are still in the process of being built. The result is many
buildkite agents doing nothing but blocking on other derivations being
built. We can easily prevent this by using the dependency information we
can get from the derivation (files) of the targets we want to build and
translating them into buildkite step dependencies.
The hard part of this has already been done for a while:
//nix/dependency-analyzer finds the dependencies between a list of
“known” derivations (even if they only depend on each other through
intermediate derivations) without depending on a specific derivation
builder convention, but rather relying on `.drv` files. It still has a
few rough edges, but has been working reliably for our purposes.
Since our steps are identified by derivation hashes, we can just
directly use the available dependency data. Luckily, buildkite seems to
just takes a step as if it was completed if it is skipped, so we don't
even have to check whether dependencies have been skipped or not.
On whitby it seems that the dependency analysis costs about a minute
additionally (which is how long it takes to run
//nix/dependency-analyzer in isolation just about).
Supersedes cl/5063, cl/5060, cl/5064 and cl/5065.
Change-Id: I91d2eb2b43d60811cac0d26fa94467298f622970
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11116
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
This will make it easier to emit intra pipeline dependencies based on
the dependencies between derivations contained therein later.
A consequence of this change is that it is no longer possible to have
the same derivation be exposed as multiple steps in the pipeline. I
doubt that having this is very useful, though.
Keys for extraSteps are not changed significantly, but are distinguished
from derivation based steps (with prefix `drv-*`) by their prefix
`extra-step-*`.
Change-Id: I4165900e512b3967fa3ca4cd5bffd44bc15915fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11115
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
Looking for .drv file names in non .drv files doesn't make sense, as it
less reliably a reference in those cases. Matches behavior of the
function for Nix >= 2.6.
Change-Id: I79fc1da3e55df869f03702fa5137d030790bc4eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11114
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: aspen <root@gws.fyi>
- Set pipefail to make sure the correct exit code gets returned.
- Don't let xargs(1) call realpath(1) if the build fails (and nothing is
printed to stdout).
Change-Id: I526a18df877601d884b83862da5f37336cdf52cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11312
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Resolves b/385. I have a feeling bufCheck should be moved to
//tools as well.
Change-Id: I2a2b63d135a2f8bcc982aa1138ff3213c6012f20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11152
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
In principle we don't want to build any (later) pipeline target during
pipeline evaluation insofar they appear in extraSteps. For this reason,
we have the needsOutput mechanism which prevents the parent target of an
extraStep from being built in 🦙.
Unfortunately, this mechanism is not general purpose enough, as we use
other (i.e. non parent) targets from depot in extraSteps. As a
consequence, kind of expensive builds need to happen during pipeline
construction at the moment. The solution is to use the fact that the
command script we want to run is exposed via the readTree interface to
depot and build the script proper only when the extra step is executed.
To facilitate this, some prerequisite changes need to be made:
- We need to use a symlink different to result in case needsOutput is
true which needs support in mkBuildCommand. We also need to avoid this
symlink being picked up by git, as many extra steps check whether the
tree is dirty or not. (Is there a way to have it outside the depot
tree?)
- Since we rely on the build command printing a single store path we
store in $command_script, we need to avoid it printing two paths
in cases where nix-store(1) is used (nix-store(1) prints the symlink
and readlink(1) would print the store path in a separate line).
Future work would be to remove/deprecate the needsOutput mechanism:
After this change the parent target wouldn't be built right away even if
it appeared in the script via string interpolation. Thus we could,
instead of expecting the target being available as `./result`, make our
extra steps nix-ier.
Change-Id: Idd2e88a865eadabe229ce1e05406e8cc4cb63f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10850
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
After the previous refactor that made mkBuildExpr use attribute paths,
mkStep and mkDrvMap can now share the code for calculating the attribute
path of any given target.
Change-Id: I50f9687e85c3b4407e4a55ff7e645db882bb80d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10849
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
mkBuild* is independent of the target abstraction, we can just use
attribute paths. That is useful because we'll want to reuse that code in
order to avoid building derivations (apart from unavoidable IfD) during
pipeline construction for extraSteps.
Breaking Change for mkBuildExpr and mkBuildCommand.
Change-Id: I03646310192087d3e50f358a714472d1ac1a652f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10848
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
When changing the buildkite pipeline code I found that some functions
have a lot of arguments where the order is not necessarily clear. To
ease further refactors / new features, I've ported them over to taking
attribute sets.
Note that this technically is a breaking change, as these functions are
all exposed. Not sure how often they'd be called from the outside,
though.
Change-Id: I118c8c5242922403d12f6e5a61beaf68f636b40a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10847
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
As far as I can tell we can handle files and directories using the same
cp(1) invocation, so we no longer need to potentially IfD derivations in
the tree to figure out whether they are files or directories.
Change-Id: Iabe648c30a747fa42768558715e388552024764a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10996
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Add //nix/writeTree, a function to make a derivation to build a
directory structure from a Nix attribute set.
Co-authored-by: sterni <sternenseemann@systemli.org>
Change-Id: I9c0fc91611a55a20ad33de6f2b27abde4b6abd21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10963
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: aspen <root@gws.fyi>
https://github.com/golang/go/issues/51225 and other changes mean that importcfgs are now basically required for Go 1.20+; we also separately compile the Go stdlib, since it looks like pkgs.go no longer actually has the compiled version of the stdlib shipped, just the source.
Change-Id: Ibf5ee7d43f7800c6dd1e0dec6c7a6d35ef50b7b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10801
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is required because Go 1.18 is actually being deleted. I've
applied the formatting breakage that it introduces (such as breaking
comment formatting), because I can't be bothered to try and work
around broken Go stuff.
Change-Id: Ica7cee0d01228845d6a766079fef36df99a3da96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9832
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
With this change, the query we make to Buildkite changes from
explicitly querying for a specific number of *ancestor* builds, to
any latest builds of the default branch that have a drvmap.
This is not really supported by Buildkite, it seems, and the query to
do it feels very wonky and requires a lot of fiddling with `jq` to get
the output into the right shape.
We lose the information about which build we downloaded this from in
the output. Adding that information back would make the `jq` query
much more complex.
Change-Id: I9e7cecdffa9ac09f9e0339eb24d98c0e8dd82292
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9168
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
buf lint already happens in other individual targets, and as no
generation happens in here anymore either, there's no need to check for
git status to record changes anymore.
Change-Id: Ieb8fc4760a61dd0f0e03f0ec388062dd2303c37a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9789
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Similar to the castore-go CL before, this also updates the store-go
bindings to the new layout.
Change-Id: Id73d7ad43f7d70171ab021728e303300c5db71f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9788
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Have `tvix/castore/protos` only contain the protos, no go noise.
Make the `.pb.go` file generation a pure Nix build
at `//tvix/castore/protos:go-bindings`, and have a script at
`//tvix:castore-go-generate` (TBD) that copies the results to
`tvix/castore-go`.
`//tvix:castore-go`, with sources in `tvix/castore-go` now contains the
tooling around the generated bindings, and the generated bindings
themselves (So go mod replace workflows still work).
An additional CI step is added from there to ensure idempotenty of
the .pb.go files.
The code.tvl.fyi webserver config is updated to the new source code
path. I'm still unsure if we want to also update the go.mod name. While
being a backwards-incompatible change, it'll probbaly make it easier
where to find these files, and the amount of external consumers is still
low enough.
Part of b/323.
Change-Id: I2edadd118c22ec08e57c693f6cc2ef3261c62489
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9787
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Without the examples, some behaviour of the merge operator might not
be clear from the previous description. Due to how pervasively the
operator is used, I think it warrants a separate section.
This fixes https://github.com/tazjin/nix-1p/issues/16
Change-Id: Iecba5f1cb749bef0a4987b3fc5642832a92c18d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9599
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
I found some links to nix manual are not working (they redirect me to
manual homepage and one github link led to 404 page), so I fixed all
the links for which I was able to find appropriate target.
Imported from https://github.com/tazjin/nix-1p/pull/15
Change-Id: Ie933951e9743649d7794a6bd2a82c3a0ed328336
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9501
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.
Notable changes:
- `fixtures` and `utils` had to be moved out of the `test` cfg, so they
can be imported from tvix-store.
- Some ad-hoc fixtures in the test were moved to proper fixtures in the
same step.
- The protos are now created by a (more static) recipe in the protos/
directory.
The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)
Bug: https://b.tvl.fyi/issues/301
Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Makes it possible to disable tag filtering for rendered content (on by
default, of course).
Change-Id: I74ecfee97eaa7abf32049172b28705e7a3f21548
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9276
Tested-by: BuildkiteCI
Reviewed-by: Mark Shevchenko <markshevchenko@gmail.com>
This allows pinning the name of the sparse tree derivation, which
stops the continous rebuilding of tvix-store-proto dependents.
I've opted to let the function take an attribute set instead and
refactored the call sites appropriately.
Change-Id: I3e57785094b1adbfffa24caf9f1c3384844fa200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8965
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This fixes https://github.com/tazjin/nix-1p/issues/13
Due to with's use of a semicolon instead of some kind of bracing
statement, people can get confused about what the scope of a `with`
is.
Hopefully this change should make it more explicit.
Change-Id: Ifaff28e09a2e8b8b244a9c11d44b2780dee284ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8951
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
