maurice/snix
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(web): migrate language-spec,lang-version,value-pointer-equality

Browse source 
Change-Id: I2008d4d5d92dc02f3955828ba93f748282948f07
Reviewed-on: https://cl.snix.dev/c/snix/+/30447
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Vova Kryachko <v.kryachko@gmail.com>
Tested-by: besadii
This commit is contained in:
Florian Klink 2025-05-04 19:34:17 +03:00 • committed by clbot
parent 52e7b5b485
commit 21628f7ad4
5 changed files with 49 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

11
web/content/docs/reference/nix-language/_index.md Normal file
View file

@ -0,0 +1,11 @@
---
title: Nix Language
slug: nix-language
description: ""
summary: ""
date: 2025-05-01T13:10:37+02:00
lastmod: 2025-05-01T13:10:37+02:00
draft: false
weight: 50
toc: true
---

72
web/content/docs/reference/nix-language/lang-version.md Normal file
View file

@ -0,0 +1,72 @@
---
title: Version history
slug: lang-version
description: ""
summary: ""
date: 2025-05-01T13:10:37+02:00
lastmod: 2025-05-01T13:10:37+02:00
draft: false
weight: 50
toc: false
---
The Nix language (“Nix”) has its own versioning mechanism independent from its
most popular implementation (“C++ Nix”): `builtins.langVersion`. It has been
increased whenever the language has changed syntactically or semantically in a
way that would not be introspectable otherwise. In particular, this does not
include addition (or removal) of `builtins`, as this can be introspected using
standard attribute set operations.
Changes to `builtins.langVersion` are best found by viewing the git history of
C++ Nix using `git log -G 'mkInt\\(v, [0-9]\\)'` for `builtins.langVersion` < 7.
After that point `git log -G 'v\\.mkInt\\([0-9]+\\)'` should work. To reduce the
amount of false positives, specify the version number you are interested in
explicitly.
## 1
The first version of the Nix language is its state at the point when
`builtins.langVersion` was added in [8b8ee53] which was first released
as part of C++ Nix 1.2.
## 2
Nix version 2 changed the behavior of `builtins.storePath`: It would now [try to
substitute the given path if missing][storePath-substitute], instead of creating
an evaluation failure. `builtins.langVersion` was increased in [e36229d].
## 3
Nix version 3 changed the behavior of the `==` behavior. Strings would now be
considered [equal even if they had differing string context][equal-no-ctx].
## 4
Nix version 4 [added the float type][float] to the language.
## 5
The [increase of `builtins.langVersion` to 5][langVersion-5] did not signify a
language change, but added support for structured attributes to the Nix daemon.
Eelco Dolstra writes as to what changed:
> The structured attributes support. Unfortunately that's not so much a language
> change as a build.cc (i.e. daemon) change, but we don't really have a way to
> express that...
Maybe `builtins.nixVersion` (which was added in version 1) should have been
used instead. In any case, the [only `langVersion` check][nixpkgs-langVersion-5]
in nixpkgs verifies a lower bound of 5.
## 6
Nix version 6 added support for [comparing two lists][list-comparison].
[8b8ee53]: https://github.com/nixos/nix/commit/8b8ee53bc73769bb25d967ba259dabc9b23e2e6f
[storePath-substitute]: https://github.com/nixos/nix/commit/22d665019a3770148929b7504c73bcdbe025ec12
[e36229d]: https://github.com/nixos/nix/commit/e36229d27f9ab508e0abf1892f3e8c263d2f8c58
[equal-no-ctx]: https://github.com/nixos/nix/commit/ee7fe64c0ac00f2be11604a2a6509eb86dc19f0a
[float]: https://github.com/nixos/nix/commit/14ebde52893263930cdcde1406cc91cc5c42556f
[langVersion-5]: https://github.com/nixos/nix/commit/8191992c83bf4387b03c5fdaba818dc2b520462d
[list-comparison]: https://github.com/nixos/nix/commit/09471d2680292af48b2788108de56a8da755d661
[nixpkgs-langVersion-5]: https://github.com/NixOS/nixpkgs/blob/d7ac3423d321b8b145ccdd1aed9dfdb280f5e391/pkgs/build-support/closure-info.nix#L11

79
web/content/docs/reference/nix-language/language-spec.md Normal file
View file

@ -0,0 +1,79 @@
---
title: Specification of the Nix Language
slug: language-spec
description: ""
summary: ""
date: 2025-05-01T13:10:37+02:00
lastmod: 2025-05-01T13:10:37+02:00
draft: false
weight: 50
toc: false
---
{{< callout context="caution" title="Caution" icon="outline/alert-triangle" >}}
This document is a work in progress. Please keep an eye on
[`topic:nix-spec`](https://cl.snix.dev/q/topic:nix-spec) for ongoing
CLs.
{{</callout>}}
Nix is a general-purpose, functional programming language which this
document aims to describe.
## Background
Nix was designed and implemented as part of the [Nix package
manager](https://nixos.org/nix). It is primarily used for generating
so-called [*derivations*](#derivations), which are data structures
describing how to build a package.
The language has been described in the
[thesis](https://edolstra.github.io/pubs/phd-thesis.pdf) introducing
the package manager, but only on a high-level. At the time of writing,
Nix is informally specified (via its only complete implementation in
the package manager) and there is no complete overview over its -
sometimes surprising - semantics.
The primary project written in Nix is
[nixpkgs](https://github.com/NixOS/nixpkgs/). Uncertainties in the
process of writing this specification are resolved by investigating
patterns in nixpkgs, which we consider canonical. The code in nixpkgs
uses a reasonable subset of the features exposed by the current
implementation, some of which are *accidental*, and is thus more
useful for specifying how the language should work.
## Introduction to Nix
Nix is a general-purpose, partially lazy, functional programming
language which provides higher-order functions, type reflection,
primitive data types such as integers, strings and floats, and
compound data structures such as lists and attribute sets.
Nix has syntactic sugar for common operations, such as those for
attribute sets, and also provides a wide range of built-in functions
which have organically accumulated over time.
Nix has a variety of legacy features that are not in practical use,
but are documented in sections of this specification for the sake of
completeness.
This document describes the syntax and abstract semantics of the Nix
language, but leaves out implementation details about how Nix can be
interpreted/compiled/analysed etc.
### Program structure
This section describes the semantic structure of Nix, and how it
relates to the rest of the specification.
Each Nix program is a single [*expression*](#expressions) denoting a
[*value*](#values) (commonly a [*function*](#functions)). Each value
has a [*type*](#types), however this type is not statically known.
Nix code is modularised through the use of the
[*import*](#builtins-import) built-in function. No separate module
system exists.
In addition to chapters describing the building blocks mentioned
above, this specificiation also describes the [*syntax*](#syntax), the
available [built-in functions](#builtins), [*error handling*](#errors)
and known [*deficiencies*](#deficiencies) in the language.

350
web/content/docs/reference/nix-language/value-pointer-equality.md Normal file
View file

@ -0,0 +1,350 @@
---
title: Value Pointer Equality
slug: value-pointer-equality
description: ""
summary: ""
date: 2025-05-01T13:10:37+02:00
lastmod: 2025-05-01T13:10:37+02:00
draft: false
weight: 55
toc: false
---
## Introduction
It is a piece of semi-obscure Nix trivia that while functions are generally not
comparable, they can be compared in certain situations. This is actually quite an
important fact, as it is essential for the evaluation of nixpkgs: The attribute sets
used to represent platforms in nixpkgs, like `stdenv.buildPlatform`, contain functions,
such as `stdenv.buildPlatform.canExecute`. When writing cross logic, one invariably
ends up writing expressions that compare these sets, e.g. `stdenv.buildPlatform !=
stdenv.hostPlatform`. Since attribute set equality is the equality of their attribute
names and values, we also end up comparing the functions within them. We can summarize
the relevant part of this behavior for platform comparisons in the following (true)
Nix expressions:
* `stdenv.hostPlatform.canExecute != stdenv.hostPlatform.canExecute`
* `stdenv.hostPlatform == stdenv.hostPlatform`
This fact is commonly referred to as pointer equality of functions (or function pointer
equality) which is not an entirely accurate name, as we'll see. This account of the
behavior states that, while functions are incomparable in general, they are comparable
insofar, as they occupy the same spot in an attribute set.
However, [a maybe lesser known trick][puck-issue] is to write a function such as the
following to allow comparing functions:
```nix
let
pointerEqual = lhs: rhs: { x = lhs; } == { x = rhs; };
f = name: "Hello, my name is ${name}";
g = name: "Hello, my name is ${name}";
in
[
(pointerEqual f f) # => true
(pointerEqual f g) # => false
]
```
Here, clearly, the function is not contained at the same position in one and the same
attribute set, but at the same position in two entirely different attribute sets. We can
also see that we are not comparing the functions themselves (e.g. their AST), but
rather if they are the same individual value (i.e. pointer equal).
To figure out the _actual_ semantics, we'll first have a look at how value (pointer) equality
works in C++ Nix, the only production ready Nix implementation currently available.
## Nix (Pointer) Equality in C++ Nix
{{< callout context="note" title="Note" icon="outline/info-circle" >}}
The summary presented here is up-to-date as of 2023-06-27 and was tested with
Nix 2.3, 2.11 and 2.15.
{{</callout>}}
### `EvalState::eqValues` and `ExprOpEq::eval`
The function implementing equality in C++ Nix is `EvalState::eqValues` which starts with
[the following bit of code][eqValues-pointer-eq]:
```cpp
bool EvalState::eqValues(Value & v1, Value & v2)
{
forceValue(v1);
forceValue(v2);
/* !!! Hack to support some old broken code that relies on pointer
equality tests between sets. (Specifically, builderDefs calls
uniqList on a list of sets.) Will remove this eventually. */
if (&v1 == &v2) return true;
```
So this immediately looks more like pointer equality of arbitrary *values* instead of functions. In fact
there is [no special code facilitating function equality][eqValues-function-eq]:
```cpp
/* Functions are incomparable. */
case nFunction:
return false;
```
So one takeaway of this is that pointer equality is neither dependent on functions nor attribute sets.
In fact, we can also write our `pointerEqual` function as:
```nix
lhs: rhs: [ lhs ] == [ rhs ]
```
It's interesting that `EvalState::eqValues` forces the left and right-hand value before trying pointer
equality. It explains that `let x = throw ""; in x == x` does not evaluate successfully, but it is puzzling why
`let f = x: x; in f == f` does not return `true`. In fact, why do we need to wrap the values in a list or
attribute set at all for our `pointerEqual` function to work?
The answer lies in [the code that evaluates `ExprOpEq`][ExprOpEq],
i.e. an expression involving the `==` operator:
```cpp
void ExprOpEq::eval(EvalState & state, Env & env, Value & v)
{
Value v1; e1->eval(state, env, v1);
Value v2; e2->eval(state, env, v2);
v.mkBool(state.eqValues(v1, v2));
}
```
As you can see, two _distinct_ `Value` structs are created, so they can never be pointer equal even
if the `union` inside points to the same bit of memory. We can thus understand what actually happens
when we check the equality of an attribute set (or list), by looking at the following expression:
```nix
let
x = { name = throw "nameless"; };
in
x == x # => causes an evaluation error
```
Because `x` can't be pointer equal, as it'll end up in the distinct structs `v1` and `v2`, it needs to be compared
by value. For this reason, the `name` attribute will be forced and an evaluation error caused.
If we rewrite the expression to use…
```nix
{ inherit x; } == { inherit x; } # => true
```
…, it'll work: The two attribute sets are compared by value, but their `x` attribute turns out to be pointer
equal _after_ forcing it. This does not throw, since forcing an attribute set does not force its attributes'
values (as forcing a list doesn't force its elements).
As we have seen, pointer equality can not only be used to compare function values, but also other
otherwise incomparable values, such as lists and attribute sets that would cause an evaluation
error if they were forced recursively. We can even switch out the `throw` for an `abort`. The limitation is
of course that we need to use a value that behaves differently depending on whether it is forced
“normally” (think `builtins.seq`) or recursively (think `builtins.deepSeq`), so thunks will generally be
evaluated before pointer equality can kick into effect.
### Other Comparisons
The `!=` operator uses `EvalState::eqValues` internally as well, so it behaves exactly as `!(a == b)`.
The `>`, `<`, `>=` and `<=` operators all desugar to [CompareValues][]
eventually which generally looks at the value type before comparing. It does,
however, rely on `EvalState::eqValues` for list comparisons
([introduced in Nix 2.5][nix-2.5-changelog]), so it is possible to compare lists
with e.g. functions in them, as long as they are equal by pointer:
```nix
let
f = x: x + 42;
in
[
([ f 2 ] > [ f 1 ]) # => true
([ f 2 ] > [ (x: x) 1]) # => error: cannot compare a function with a function
([ f ] > [ f ]) # => false
]
```
Finally, since `builtins.elem` relies on `EvalState::eqValues`, you can check for
a function by pointer equality:
```nix
let
f = x: f x;
in
builtins.elem f [ f 2 3 ] # => true
```
### Pointer Equality Preserving Nix Operations
We have seen that pointer equality is established by comparing the memory
location of two C++ `Value` structs. But how does this _representation_ relate
to Nix values _themselves_ (in the sense of a platonic ideal if you will)? In
Nix, values have no identity (ignoring `unsafeGetAttrPos`) or memory location.
Since Nix is purely functional, values can't be mutated, so they need to be
copied frequently. With Nix being garbage collected, there is no strong
expectation when a copy is made, we probably just hope it is done as seldomly as
possible to save on memory. With pointer equality leaking the memory location of
the `Value` structs to an extent, it is now suddenly our business to know
exactly _when_ a copy of a value is made.
Evaluation in C++ Nix mainly proceeds along the following [two
functions][eval-maybeThunk].
```cpp
struct Expr
{
/* … */
virtual void eval(EvalState & state, Env & env, Value & v);
virtual Value * maybeThunk(EvalState & state, Env & env);
/* … */
};
```
As you can see, `Expr::eval` always takes a reference to a struct _allocated by
the caller_ to place the evaluation result in. Anything that is processed using
`Expr::eval` will be a copy of the `Value` struct even if the value before and
after are the same.
`Expr::maybeThunk`, on the other hand, returns a pointer to a `Value` which may
already exist or be newly allocated. So, if evaluation passes through `maybeThunk`,
Nix values _can_ retain their pointer equality. Since Nix is lazy, a lot of
evaluation needs to be thunked and pass through `maybeThunk`—knowing under what
circumstances `maybeThunk` will return a pointer to an already existing `Value`
struct thus means knowing the circumstances under which pointer equality of a
Nix value will be preserved in C++ Nix.
The [default case][maybeThunk-default] of `Expr::maybeThunk` allocates a new
`Value` which holds the delayed computation of the `Expr` as a thunk:
```cpp
Value * Expr::maybeThunk(EvalState & state, Env & env)
{
Value * v = state.allocValue();
mkThunk(*v, env, this);
return v;
}
```
Consequently, only special cased expressions could preserve pointer equality.
These are `ExprInt`, `ExprFloat`, `ExprString`, `ExprPath`—all of which relate
to creating new values—and [finally, `ExprVar`][maybeThunk-ExprVar]:
```cpp
Value * ExprVar::maybeThunk(EvalState & state, Env & env)
{
Value * v = state.lookupVar(&env, *this, true);
/* The value might not be initialised in the environment yet.
In that case, ignore it. */
if (v) { state.nrAvoided++; return v; }
return Expr::maybeThunk(state, env);
}
```
Here we may actually return an already existing `Value` struct. Consequently,
accessing a value from the scope is the only thing you can do with a value in
C++ Nix that preserves its pointer equality, as the following example shows:
For example, using the select operator to get a value from an attribute set
or even passing a value trough the identity function invalidates its pointer
equality to itself (or rather, its former self).
```nix
let
pointerEqual = a: b: [ a ] == [ b ];
id = x: x;
f = _: null;
x = { inherit f; };
y = { inherit f; };
in
[
(pointerEqual f f) # => true
(pointerEqual f (id f)) # => false
(pointerEqual x.f y.f) # => false
(pointerEqual x.f x.f) # => false
(pointerEqual x x) # => true
(pointerEqual x y) # => true
]
```
In the last two cases, the example also shows that there is another way to
preserve pointer equality: Storing a value in an attribute set (or list)
preserves its pointer equality even if the structure holding it is modified in
some way (as long as the value we care about is left untouched). The catch is,
of course, that there is no way to get the value out of the structure while
preserving pointer equality (which requires using the select operator or a call
to `builtins.elemAt`).
We initially illustrated the issue of pointer equality using the following
true expressions:
* `stdenv.hostPlatform.canExecute != stdenv.hostPlatform.canExecute`
* `stdenv.hostPlatform == stdenv.hostPlatform`
We can now add a third one, illustrating that pointer equality is invalidated
by select operations:
* `[ stdenv.hostPlatform.canExecute ] != [ stdenv.hostPlatform.canExecute ]`
To summarize, pointer equality is established on the memory location of the
`Value` struct in C++ Nix. Except for simple values (`int`, `bool`, …),
the `Value` struct only consists of a pointer to the actual representation
of the value (attribute set, list, function, …) and is thus cheap to copy.
In practice, this happens when a value passes through the evaluation of
almost any Nix expression. Only in the select cases described above
a value preserves its pointer equality despite being unchanged by an
expression. We can call this behavior *exterior pointer equality*.
## Summary
When comparing two Nix values, we must force both of them (non-recursively!), but are
allowed to short-circuit the comparison based on pointer equality, i.e. if they are at
the same exact value in memory, they are deemed equal immediately. This is completely
independent of what type of value they are. If they are not pointer equal, they are
(recursively) compared by value as expected.
However, when evaluating the Nix expression `a == b`, we *must* invoke our implementation's
value equality function in a way that `a` and `b` themselves can never be deemed pointer equal.
Any values we encounter while recursing during the equality check must be compared by
pointer as described above, though.
## Stability of the Feature
Keen readers will have noticed the following comment in the C++ Nix source code,
indicating that pointer comparison may be removed in the future.
```cpp
/* !!! Hack to support some old broken code that relies on pointer
equality tests between sets. (Specifically, builderDefs calls
uniqList on a list of sets.) Will remove this eventually. */
```
Now, I can't speak for the upstream C++ Nix developers, but sure can speculate.
As already pointed out, this feature is currently needed for evaluating nixpkgs.
While its use could realistically be eliminated (only bothersome spot is probably
the `emulator` function, but that should also be doable), removing the feature
would seriously compromise C++ Nix's ability to evaluate historical nixpkgs
revision which is arguably a strength of the system.
Another indication that it is likely here to stay is that it has already
[outlived builderDefs][], even though
it was (apparently) reintroduced just for this use case. More research into
the history of this feature would still be prudent, especially the reason for
its original introduction (maybe performance?).
[puck-issue]: https://github.com/NixOS/nix/issues/3371
[eqValues-pointer-eq]: https://github.com/NixOS/nix/blob/3c618c43c6044eda184df235c193877529e951cb/src/libexpr/eval.cc#L2401-L2404
[eqValues-function-eq]: https://github.com/NixOS/nix/blob/3c618c43c6044eda184df235c193877529e951cb/src/libexpr/eval.cc#L2458-L2460
[ExprOpEq]: https://github.com/NixOS/nix/blob/3c618c43c6044eda184df235c193877529e951cb/src/libexpr/eval.cc#L1822-L1827
[outlived builderDefs]: https://github.com/NixOS/nixpkgs/issues/4210
[CompareValues]: https://github.com/NixOS/nix/blob/3c618c43c6044eda184df235c193877529e951cb/src/libexpr/primops.cc#L569-L610
[nix-2.5-changelog]: https://nixos.org/manual/nix/stable/release-notes/rl-2.5.html
[eval-maybeThunk]: https://github.com/NixOS/nix/blob/3c618c43c6044eda184df235c193877529e951cb/src/libexpr/nixexpr.hh#L161-L162
[maybeThunk-default]: https://github.com/NixOS/nix/blob/8e770dac9f68162cfbb368e53f928df491babff3/src/libexpr/eval.cc#L1076-L1081
[maybeThunk-ExprVar]: https://github.com/NixOS/nix/blob/8e770dac9f68162cfbb368e53f928df491babff3/src/libexpr/eval.cc#L1084-L1091