maurice/snix
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add 'third_party/nix/' from commit 'be66c7a6b24e3c3c6157fd37b86c7203d14acf10'

Browse source 
git-subtree-dir: third_party/nix
git-subtree-mainline: cf8cd640c1
git-subtree-split: be66c7a6b24e3c3c6157fd37b86c7203d14acf10
This commit is contained in:
Vincent Ambo 2020-05-17 15:52:38 +01:00
commit 7994fd1d54
737 changed files with 105390 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

14
third_party/nix/doc/manual/advanced-topics/advanced-topics.xml vendored Normal file
View file

@ -0,0 +1,14 @@
<part xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="part-advanced-topics"
version="5.0">
<title>Advanced Topics</title>
<xi:include href="distributed-builds.xml" />
<xi:include href="cores-vs-jobs.xml" />
<xi:include href="diff-hook.xml" />
<xi:include href="post-build-hook.xml" />
</part>

121
third_party/nix/doc/manual/advanced-topics/cores-vs-jobs.xml vendored Normal file
View file

@ -0,0 +1,121 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="chap-tuning-cores-and-jobs">
<title>Tuning Cores and Jobs</title>
<para>Nix has two relevant settings with regards to how your CPU cores
will be utilized: <xref linkend="conf-cores" /> and
<xref linkend="conf-max-jobs" />. This chapter will talk about what
they are, how they interact, and their configuration trade-offs.</para>
<variablelist>
<varlistentry>
<term><xref linkend="conf-max-jobs" /></term>
<listitem><para>
Dictates how many separate derivations will be built at the same
time. If you set this to zero, the local machine will do no
builds. Nix will still substitute from binary caches, and build
remotely if remote builders are configured.
</para></listitem>
</varlistentry>
<varlistentry>
<term><xref linkend="conf-cores" /></term>
<listitem><para>
Suggests how many cores each derivation should use. Similar to
<command>make -j</command>.
</para></listitem>
</varlistentry>
</variablelist>
<para>The <xref linkend="conf-cores" /> setting determines the value of
<envar>NIX_BUILD_CORES</envar>. <envar>NIX_BUILD_CORES</envar> is equal
to <xref linkend="conf-cores" />, unless <xref linkend="conf-cores" />
equals <literal>0</literal>, in which case <envar>NIX_BUILD_CORES</envar>
will be the total number of cores in the system.</para>
<para>The total number of consumed cores is a simple multiplication,
<xref linkend="conf-cores" /> * <envar>NIX_BUILD_CORES</envar>.</para>
<para>The balance on how to set these two independent variables depends
upon each builder's workload and hardware. Here are a few example
scenarios on a machine with 24 cores:</para>
<table>
<caption>Balancing 24 Build Cores</caption>
<thead>
<tr>
<th><xref linkend="conf-max-jobs" /></th>
<th><xref linkend="conf-cores" /></th>
<th><envar>NIX_BUILD_CORES</envar></th>
<th>Maximum Processes</th>
<th>Result</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>24</td>
<td>24</td>
<td>24</td>
<td>
One derivation will be built at a time, each one can use 24
cores. Undersold if a job can’t use 24 cores.
</td>
</tr>
<tr>
<td>4</td>
<td>6</td>
<td>6</td>
<td>24</td>
<td>
Four derivations will be built at once, each given access to
six cores.
</td>
</tr>
<tr>
<td>12</td>
<td>6</td>
<td>6</td>
<td>72</td>
<td>
12 derivations will be built at once, each given access to six
cores. This configuration is over-sold. If all 12 derivations
being built simultaneously try to use all six cores, the
machine's performance will be degraded due to extensive context
switching between the 12 builds.
</td>
</tr>
<tr>
<td>24</td>
<td>1</td>
<td>1</td>
<td>24</td>
<td>
24 derivations can build at the same time, each using a single
core. Never oversold, but derivations which require many cores
will be very slow to compile.
</td>
</tr>
<tr>
<td>24</td>
<td>0</td>
<td>24</td>
<td>576</td>
<td>
24 derivations can build at the same time, each using all the
available cores of the machine. Very likely to be oversold,
and very likely to suffer context switches.
</td>
</tr>
</tbody>
</table>
<para>It is up to the derivations' build script to respect
host's requested cores-per-build by following the value of the
<envar>NIX_BUILD_CORES</envar> environment variable.</para>
</chapter>

205
third_party/nix/doc/manual/advanced-topics/diff-hook.xml vendored Normal file
View file

@ -0,0 +1,205 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="chap-diff-hook"
version="5.0"
>
<title>Verifying Build Reproducibility with <option linkend="conf-diff-hook">diff-hook</option></title>
<subtitle>Check build reproducibility by running builds multiple times
and comparing their results.</subtitle>
<para>Specify a program with Nix's <xref linkend="conf-diff-hook" /> to
compare build results when two builds produce different results. Note:
this hook is only executed if the results are not the same, this hook
is not used for determining if the results are the same.</para>
<para>For purposes of demonstration, we'll use the following Nix file,
<filename>deterministic.nix</filename> for testing:</para>
<programlisting>
let
inherit (import &lt;nixpkgs&gt; {}) runCommand;
in {
stable = runCommand "stable" {} ''
touch $out
'';
unstable = runCommand "unstable" {} ''
echo $RANDOM > $out
'';
}
</programlisting>
<para>Additionally, <filename>nix.conf</filename> contains:
<programlisting>
diff-hook = /etc/nix/my-diff-hook
run-diff-hook = true
</programlisting>
where <filename>/etc/nix/my-diff-hook</filename> is an executable
file containing:
<programlisting>
#!/bin/sh
exec &gt;&amp;2
echo "For derivation $3:"
/run/current-system/sw/bin/diff -r "$1" "$2"
</programlisting>
</para>
<para>The diff hook is executed by the same user and group who ran the
build. However, the diff hook does not have write access to the store
path just built.</para>
<section>
<title>
Spot-Checking Build Determinism
</title>
<para>
Verify a path which already exists in the Nix store by passing
<option>--check</option> to the build command.
</para>
<para>If the build passes and is deterministic, Nix will exit with a
status code of 0:</para>
<screen>
$ nix-build ./deterministic.nix -A stable
these derivations will be built:
/nix/store/z98fasz2jqy9gs0xbvdj939p27jwda38-stable.drv
building '/nix/store/z98fasz2jqy9gs0xbvdj939p27jwda38-stable.drv'...
/nix/store/yyxlzw3vqaas7wfp04g0b1xg51f2czgq-stable
$ nix-build ./deterministic.nix -A stable --check
checking outputs of '/nix/store/z98fasz2jqy9gs0xbvdj939p27jwda38-stable.drv'...
/nix/store/yyxlzw3vqaas7wfp04g0b1xg51f2czgq-stable
</screen>
<para>If the build is not deterministic, Nix will exit with a status
code of 1:</para>
<screen>
$ nix-build ./deterministic.nix -A unstable
these derivations will be built:
/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv
building '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv'...
/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable
$ nix-build ./deterministic.nix -A unstable --check
checking outputs of '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv'...
error: derivation '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv' may not be deterministic: output '/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable' differs
</screen>
<para>In the Nix daemon's log, we will now see:
<screen>
For derivation /nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv:
1c1
&lt; 8108
---
&gt; 30204
</screen>
</para>
<para>Using <option>--check</option> with <option>--keep-failed</option>
will cause Nix to keep the second build's output in a special,
<literal>.check</literal> path:</para>
<screen>
$ nix-build ./deterministic.nix -A unstable --check --keep-failed
checking outputs of '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv'...
note: keeping build directory '/tmp/nix-build-unstable.drv-0'
error: derivation '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv' may not be deterministic: output '/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable' differs from '/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable.check'
</screen>
<para>In particular, notice the
<literal>/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable.check</literal>
output. Nix has copied the build results to that directory where you
can examine it.</para>
<note xml:id="check-dirs-are-unregistered">
<title><literal>.check</literal> paths are not registered store paths</title>
<para>Check paths are not protected against garbage collection,
and this path will be deleted on the next garbage collection.</para>
<para>The path is guaranteed to be alive for the duration of
<xref linkend="conf-diff-hook" />'s execution, but may be deleted
any time after.</para>
<para>If the comparison is performed as part of automated tooling,
please use the diff-hook or author your tooling to handle the case
where the build was not deterministic and also a check path does
not exist.</para>
</note>
<para>
<option>--check</option> is only usable if the derivation has
been built on the system already. If the derivation has not been
built Nix will fail with the error:
<screen>
error: some outputs of '/nix/store/hzi1h60z2qf0nb85iwnpvrai3j2w7rr6-unstable.drv' are not valid, so checking is not possible
</screen>
Run the build without <option>--check</option>, and then try with
<option>--check</option> again.
</para>
</section>
<section>
<title>
Automatic and Optionally Enforced Determinism Verification
</title>
<para>
Automatically verify every build at build time by executing the
build multiple times.
</para>
<para>
Setting <xref linkend="conf-repeat" /> and
<xref linkend="conf-enforce-determinism" /> in your
<filename>nix.conf</filename> permits the automated verification
of every build Nix performs.
</para>
<para>
The following configuration will run each build three times, and
will require the build to be deterministic:
<programlisting>
enforce-determinism = true
repeat = 2
</programlisting>
</para>
<para>
Setting <xref linkend="conf-enforce-determinism" /> to false as in
the following configuration will run the build multiple times,
execute the build hook, but will allow the build to succeed even
if it does not build reproducibly:
<programlisting>
enforce-determinism = false
repeat = 1
</programlisting>
</para>
<para>
An example output of this configuration:
<screen>
$ nix-build ./test.nix -A unstable
these derivations will be built:
/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv
building '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' (round 1/2)...
building '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' (round 2/2)...
output '/nix/store/6xg356v9gl03hpbbg8gws77n19qanh02-unstable' of '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' differs from '/nix/store/6xg356v9gl03hpbbg8gws77n19qanh02-unstable.check' from previous round
/nix/store/6xg356v9gl03hpbbg8gws77n19qanh02-unstable
</screen>
</para>
</section>
</chapter>

190
third_party/nix/doc/manual/advanced-topics/distributed-builds.xml vendored Normal file
View file

@ -0,0 +1,190 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id='chap-distributed-builds'>
<title>Remote Builds</title>
<para>Nix supports remote builds, where a local Nix installation can
forward Nix builds to other machines. This allows multiple builds to
be performed in parallel and allows Nix to perform multi-platform
builds in a semi-transparent way. For instance, if you perform a
build for a <literal>x86_64-darwin</literal> on an
<literal>i686-linux</literal> machine, Nix can automatically forward
the build to a <literal>x86_64-darwin</literal> machine, if
available.</para>
<para>To forward a build to a remote machine, it’s required that the
remote machine is accessible via SSH and that it has Nix
installed. You can test whether connecting to the remote Nix instance
works, e.g.
<screen>
$ nix ping-store --store ssh://mac
</screen>
will try to connect to the machine named <literal>mac</literal>. It is
possible to specify an SSH identity file as part of the remote store
URI, e.g.
<screen>
$ nix ping-store --store ssh://mac?ssh-key=/home/alice/my-key
</screen>
Since builds should be non-interactive, the key should not have a
passphrase. Alternatively, you can load identities ahead of time into
<command>ssh-agent</command> or <command>gpg-agent</command>.</para>
<para>If you get the error
<screen>
bash: nix-store: command not found
error: cannot connect to 'mac'
</screen>
then you need to ensure that the <envar>PATH</envar> of
non-interactive login shells contains Nix.</para>
<warning><para>If you are building via the Nix daemon, it is the Nix
daemon user account (that is, <literal>root</literal>) that should
have SSH access to the remote machine. If you can’t or don’t want to
configure <literal>root</literal> to be able to access to remote
machine, you can use a private Nix store instead by passing
e.g. <literal>--store ~/my-nix</literal>.</para></warning>
<para>The list of remote machines can be specified on the command line
or in the Nix configuration file. The former is convenient for
testing. For example, the following command allows you to build a
derivation for <literal>x86_64-darwin</literal> on a Linux machine:
<screen>
$ uname
Linux
$ nix build \
'(with import &lt;nixpkgs> { system = "x86_64-darwin"; }; runCommand "foo" {} "uname > $out")' \
--builders 'ssh://mac x86_64-darwin'
[1/0/1 built, 0.0 MiB DL] building foo on ssh://mac
$ cat ./result
Darwin
</screen>
It is possible to specify multiple builders separated by a semicolon
or a newline, e.g.
<screen>
--builders 'ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd'
</screen>
</para>
<para>Each machine specification consists of the following elements,
separated by spaces. Only the first element is required.
To leave a field at its default, set it to <literal>-</literal>.
<orderedlist>
<listitem><para>The URI of the remote store in the format
<literal>ssh://[<replaceable>username</replaceable>@]<replaceable>hostname</replaceable></literal>,
e.g. <literal>ssh://nix@mac</literal> or
<literal>ssh://mac</literal>. For backward compatibility,
<literal>ssh://</literal> may be omitted. The hostname may be an
alias defined in your
<filename>~/.ssh/config</filename>.</para></listitem>
<listitem><para>A comma-separated list of Nix platform type
identifiers, such as <literal>x86_64-darwin</literal>. It is
possible for a machine to support multiple platform types, e.g.,
<literal>i686-linux,x86_64-linux</literal>. If omitted, this
defaults to the local platform type.</para></listitem>
<listitem><para>The SSH identity file to be used to log in to the
remote machine. If omitted, SSH will use its regular
identities.</para></listitem>
<listitem><para>The maximum number of builds that Nix will execute
in parallel on the machine. Typically this should be equal to the
number of CPU cores. For instance, the machine
<literal>itchy</literal> in the example will execute up to 8 builds
in parallel.</para></listitem>
<listitem><para>The “speed factor”, indicating the relative speed of
the machine. If there are multiple machines of the right type, Nix
will prefer the fastest, taking load into account.</para></listitem>
<listitem><para>A comma-separated list of <emphasis>supported
features</emphasis>. If a derivation has the
<varname>requiredSystemFeatures</varname> attribute, then Nix will
only perform the derivation on a machine that has the specified
features. For instance, the attribute
<programlisting>
requiredSystemFeatures = [ "kvm" ];
</programlisting>
will cause the build to be performed on a machine that has the
<literal>kvm</literal> feature.</para></listitem>
<listitem><para>A comma-separated list of <emphasis>mandatory
features</emphasis>. A machine will only be used to build a
derivation if all of the machine’s mandatory features appear in the
derivation’s <varname>requiredSystemFeatures</varname>
attribute..</para></listitem>
</orderedlist>
For example, the machine specification
<programlisting>
nix@scratchy.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy_auto 8 1 kvm
nix@itchy.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy_auto 8 2
nix@poochie.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy_auto 1 2 kvm benchmark
</programlisting>
specifies several machines that can perform
<literal>i686-linux</literal> builds. However,
<literal>poochie</literal> will only do builds that have the attribute
<programlisting>
requiredSystemFeatures = [ "benchmark" ];
</programlisting>
or
<programlisting>
requiredSystemFeatures = [ "benchmark" "kvm" ];
</programlisting>
<literal>itchy</literal> cannot do builds that require
<literal>kvm</literal>, but <literal>scratchy</literal> does support
such builds. For regular builds, <literal>itchy</literal> will be
preferred over <literal>scratchy</literal> because it has a higher
speed factor.</para>
<para>Remote builders can also be configured in
<filename>nix.conf</filename>, e.g.
<programlisting>
builders = ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd
</programlisting>
Finally, remote builders can be configured in a separate configuration
file included in <option>builders</option> via the syntax
<literal>@<replaceable>file</replaceable></literal>. For example,
<programlisting>
builders = @/etc/nix/machines
</programlisting>
causes the list of machines in <filename>/etc/nix/machines</filename>
to be included. (This is the default.)</para>
<para>If you want the builders to use caches, you likely want to set
the option <link linkend='conf-builders-use-substitutes'><literal>builders-use-substitutes</literal></link>
in your local <filename>nix.conf</filename>.</para>
<para>To build only on remote builders and disable building on the local machine,
you can use the option <option>--max-jobs 0</option>.</para>
</chapter>

160
third_party/nix/doc/manual/advanced-topics/post-build-hook.xml vendored Normal file
View file

@ -0,0 +1,160 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="chap-post-build-hook"
version="5.0"
>
<title>Using the <xref linkend="conf-post-build-hook" /></title>
<subtitle>Uploading to an S3-compatible binary cache after each build</subtitle>
<section xml:id="chap-post-build-hook-caveats">
<title>Implementation Caveats</title>
<para>Here we use the post-build hook to upload to a binary cache.
This is a simple and working example, but it is not suitable for all
use cases.</para>
<para>The post build hook program runs after each executed build,
and blocks the build loop. The build loop exits if the hook program
fails.</para>
<para>Concretely, this implementation will make Nix slow or unusable
when the internet is slow or unreliable.</para>
<para>A more advanced implementation might pass the store paths to a
user-supplied daemon or queue for processing the store paths outside
of the build loop.</para>
</section>
<section>
<title>Prerequisites</title>
<para>
This tutorial assumes you have configured an S3-compatible binary cache
according to the instructions at
<xref linkend="ssec-s3-substituter-authenticated-writes" />, and
that the <literal>root</literal> user's default AWS profile can
upload to the bucket.
</para>
</section>
<section>
<title>Set up a Signing Key</title>
<para>Use <command>nix-store --generate-binary-cache-key</command> to
create our public and private signing keys. We will sign paths
with the private key, and distribute the public key for verifying
the authenticity of the paths.</para>
<screen>
# nix-store --generate-binary-cache-key example-nix-cache-1 /etc/nix/key.private /etc/nix/key.public
# cat /etc/nix/key.public
example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM=
</screen>
<para>Then, add the public key and the cache URL to your
<filename>nix.conf</filename>'s <xref linkend="conf-trusted-public-keys" />
and <xref linkend="conf-substituters" /> like:</para>
<programlisting>
substituters = https://cache.nixos.org/ s3://example-nix-cache
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM=
</programlisting>
<para>we will restart the Nix daemon a later step.</para>
</section>
<section>
<title>Implementing the build hook</title>
<para>Write the following script to
<filename>/etc/nix/upload-to-cache.sh</filename>:
</para>
<programlisting>
#!/bin/sh
set -eu
set -f # disable globbing
export IFS=' '
echo "Signing paths" $OUT_PATHS
nix sign-paths --key-file /etc/nix/key.private $OUT_PATHS
echo "Uploading paths" $OUT_PATHS
exec nix copy --to 's3://example-nix-cache' $OUT_PATHS
</programlisting>
<note>
<title>Should <literal>$OUT_PATHS</literal> be quoted?</title>
<para>
The <literal>$OUT_PATHS</literal> variable is a space-separated
list of Nix store paths. In this case, we expect and want the
shell to perform word splitting to make each output path its
own argument to <command>nix sign-paths</command>. Nix guarantees
the paths will not contain any spaces, however a store path
might contain glob characters. The <command>set -f</command>
disables globbing in the shell.
</para>
</note>
<para>
Then make sure the hook program is executable by the <literal>root</literal> user:
<screen>
# chmod +x /etc/nix/upload-to-cache.sh
</screen></para>
</section>
<section>
<title>Updating Nix Configuration</title>
<para>Edit <filename>/etc/nix/nix.conf</filename> to run our hook,
by adding the following configuration snippet at the end:</para>
<programlisting>
post-build-hook = /etc/nix/upload-to-cache.sh
</programlisting>
<para>Then, restart the <command>nix-daemon</command>.</para>
</section>
<section>
<title>Testing</title>
<para>Build any derivation, for example:</para>
<screen>
$ nix-build -E '(import &lt;nixpkgs&gt; {}).writeText "example" (builtins.toString builtins.currentTime)'
these derivations will be built:
/nix/store/s4pnfbkalzy5qz57qs6yybna8wylkig6-example.drv
building '/nix/store/s4pnfbkalzy5qz57qs6yybna8wylkig6-example.drv'...
running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
post-build-hook: Signing paths /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
post-build-hook: Uploading paths /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
/nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
</screen>
<para>Then delete the path from the store, and try substituting it from the binary cache:</para>
<screen>
$ rm ./result
$ nix-store --delete /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
</screen>
<para>Now, copy the path back from the cache:</para>
<screen>
$ nix store --realize /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
copying path '/nix/store/m8bmqwrch6l3h8s0k3d673xpmipcdpsa-example from 's3://example-nix-cache'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/m8bmqwrch6l3h8s0k3d673xpmipcdpsa-example
</screen>
</section>
<section>
<title>Conclusion</title>
<para>
We now have a Nix installation configured to automatically sign and
upload every local build to a remote binary cache.
</para>
<para>
Before deploying this to production, be sure to consider the
implementation caveats in <xref linkend="chap-post-build-hook-caveats" />.
</para>
</section>
</chapter>