diff --git a/snix/docs/src/architecture.md b/snix/docs/src/architecture.md index 02ffdfdcd..5f7da9809 100644 --- a/snix/docs/src/architecture.md +++ b/snix/docs/src/architecture.md @@ -46,60 +46,6 @@ coordinator on top of a state-machine model that would make it possible to reuse the FSM logic without tying it to any particular kind of application. -### Evaluator - -*Purpose:* Eval takes care of evaluating Nix code. In a typical build -flow it would be responsible for producing derivations. It can also be -used as a standalone tool, for example, in use-cases where Nix is used -to generate configuration without any build or store involvement. - -*Requirements:* For now, it will run on the machine invoking the build -command itself. We give it filesystem access to handle things like -imports or `builtins.readFile`. - -To support IFD, the Evaluator also needs access to store paths. This -could be implemented by having the coordinator provide an interface to retrieve -files from a store path, or by ensuring a "realized version of the store" is -accessible by the evaluator (this could be a FUSE filesystem, or the "real" -/nix/store on disk. - -We might be okay with running the evaluator with filesystem access for now and -can extend the interface if the need arises. - -### Builder - -*Purpose:* A builder receives derivations from the coordinator and -builds them. - -By making builder a standardised interface it's possible to make the -sandboxing mechanism used by the build process pluggable. - -Nix is currently using a hard-coded -[libseccomp](https://github.com/seccomp/libseccomp) based sandboxing -mechanism and another one based on -[sandboxd](https://www.unix.com/man-page/mojave/8/sandboxd/) on macOS. -These are only separated by [compiler preprocessor -macros](https://gcc.gnu.org/onlinedocs/cpp/Ifdef.html) within the same -source files despite having very little in common with each other. - -This makes experimentation with alternative backends difficult and -porting Nix to other platforms harder than it has to be. We want to -write a new Linux builder which uses -[OCI](https://github.com/opencontainers/runtime-spec), the current -dominant Linux containerisation technology, by default. - -With a well-defined builder abstraction, it's also easy to imagine -other backends such as a Kubernetes-based one in the future. - -The environment in which builds happen is currently very Nix-specific. We might -want to avoid having to maintain all the intricacies of a Nix-specific -sandboxing environment in every builder, and instead only provide a more -generic interface, receiving build requests (and have the coordinator translate -derivations to that format). [^1] - -To build, the builder needs to be able to mount all build inputs into the build -environment. For this, it needs the store to expose a filesystem interface. - ### Store *Purpose:* Store takes care of storing build results. It provides a diff --git a/web/content/docs/components/build/protocol.md b/web/content/docs/components/build/protocol.md index fe66eb76b..60c0953d7 100644 --- a/web/content/docs/components/build/protocol.md +++ b/web/content/docs/components/build/protocol.md @@ -10,14 +10,50 @@ weight: 41 toc: true --- -One goal of the builder protocol is to not be too tied to the Nix implementation -itself, allowing it to be used for other builds/workloads in the future. +## Standardized Interface +One goal is to make the build protocol a standardized interface, allowing to +make the sandboxing mechanism used by the build process pluggable. -This means the builder protocol is versatile enough to express the environment a -Nix build expects, while not being aware of "what any of this means". +Nix is currently using a hard-coded [libseccomp][] based sandboxing mechanism +and another one based on [sandboxd][] on macOS. +These are only separated by [compiler preprocessor macros][ifdef] within the same +source files despite having very little in common with each other. + +In Snix, the Builders need to implement a trait, and there are multiple +implementations. In addition to an [OCI][] builder[^k8s], we also include a gRPC +client (and server adapter), allowing to run the builder both locally or +remotely, or plug in your entirely separate Builder, as long as it speaks the +same gRPC protocol. + +Check `build/protos/build.proto` for a detailed description of the protocol, +individual fields, and the tests in `glue/src/tvix_build.rs` for some examples. + +While we're somewhat confident about the `BuildRequest`, the RPC method itself +will change to a stream of events, so we can stream logs/build telemetry to the +requesting client. + +## Unaware of Nix sandbox internals + +The environment in which builds currently happen is currently very Nix-specific. +In Snix, we don't want to maintain all the intricacies of a Nix-specific +sandboxing environment in every builder, and instead only provide a more +generic interface, receiving more generic build requests (and translate +Derivations into this format). [^reapi] + +Another goal of the builder protocol is to not be too tied to the Nix +implementation itself, allowing it to be used for other builds/workloads in the +future (and experimenting with different hashing schemes etc without having to +change builder code). + + + +In concrete terms, this means the builder protocol is versatile enough to +express the environment a Nix build sets up, while it itself is not aware of +"what any of this means". For example, it is not aware of how certain environment variables are set in a -nix build, but allows specifying environment variables that should be set. +nix build, but provides the necessary infrastructure to specify environment +variables that should be set. It's also not aware of what nix store paths are. Instead, it allows: @@ -31,5 +67,37 @@ In case all specified paths are produced, and the command specified in This happens to be sufficient to *also* express how Nix builds works. -Check `build/protos/build.proto` for a detailed description of the individual -fields, and the tests in `glue/src/tvix_build.rs` for some examples. +## More hermetic builds, Build Provenance +Nix uses derivations (encoded in ATerm) as nodes in its build graph, but it +refers to other store paths used in that build by these store paths only. As +mentioned before, store paths only address the inputs - and not the content. + +This poses a big problem in Nix as soon as builds are scheduled on remote +builders: There is no guarantee that files at the same store path on the remote +builder actually have the same contents as on the machine orchestrating the +build. If a package is not binary reproducible, this can lead to so-called +[frankenbuilds]. + +This also introduces a dependency on the state that's present on the remote +builder machine: Whatever is in its store and matches the paths will be used, +even if it was maliciously placed there. + +To eliminate this hermiticity problem and increase the integrity of builds, +we've decided to use content-addressing in the builder protocol. + +In the long run, recording this information is gonna improve our posture +regarding [Build Provenance][slsa-provenance]. + + +[OCI]: https://github.com/opencontainers/runtime-spec +[libseccomp]: https://github.com/seccomp/libseccomp +[sandboxd]: https://www.unix.com/man-page/mojave/8/sandboxd/ +[ifdef]: https://gcc.gnu.org/onlinedocs/cpp/Ifdef.html +[frankenbuilds]: https://blog.layus.be/posts/2021-06-25-frankenbuilds.html +[slsa-provenance]: https://slsa.dev/spec/v1.0/provenance + +[^k8s]: With a well-defined builder abstraction, it's also easy to imagine + other backends such as a Kubernetes-based one in the future. +[^reapi]: There have already been some discussions in the Nix community, to switch + to REAPI: + https://discourse.nixos.org/t/a-proposal-for-replacing-the-nix-worker-protocol/20926/22 diff --git a/web/content/docs/components/overview.md b/web/content/docs/components/overview.md index 402f435e2..617553d72 100644 --- a/web/content/docs/components/overview.md +++ b/web/content/docs/components/overview.md @@ -53,6 +53,9 @@ in there - new formats etc. are usually "factored out into nix-compat". The builder consumes build requests from a client, runs builds and sends logs/telemetry to the client. +By making the build protocol a standardized interface, it's possible to make the +sandboxing mechanism used by the build process pluggable. + There currently exists an OCI builder, as well as gRPC server adapter and client implementations, allowing to run the builder both locally or remotely.