snix/nix/buildLisp/README.md

buildLisp.nix
=============

This is a build system for Common Lisp, written in Nix.

It aims to offer an alternative to ASDF for users who live in a
Nix-based ecosystem. This offers several advantages over ASDF:

* Simpler (almost logic-less) package definitions
* Easy linking of native dependencies (from Nix)
* Composability with Nix tooling for other languages
* Effective, per-system caching strategies
* Easy overriding of dependencies and whatnot
* Convenient support for multiple Common Lisp implementations
* ... and more!

The project is still in its early stages and some important
restrictions should be highlighted:

* Extending `buildLisp` with support for a custom implementation
  currently requires some knowledge of internals and may not be
  considered stable yet.
* Parallel compilation is not possible: Since buildLisp doesn't encode
  dependencies between components (i. e. source files) like ASDF,
  it must compile source files in sequence to avoid errors due to
  undefined symbols.

## Usage

`buildLisp` exposes four different functions:

* `buildLisp.library`: Builds a collection of Lisp files into a library.

  | parameter | type         | use                           | required? |
  |-----------|--------------|-------------------------------|-----------|
  | `name`    | `string`     | Name of the library           | yes       |
  | `srcs`    | `list<path>` | List of paths to source files | yes       |
  | `deps`    | `list<drv>`  | List of dependencies          | no        |
  | `native`  | `list<drv>`  | List of native dependencies   | no        |
  | `test`    | see "Tests"  | Specification for test suite  | no        |
  | `implementation` | see "Implementations" | Common Lisp implementation to use | no |

  The output of invoking this is a directory containing a FASL file
  that is the concatenated result of all compiled sources.

* `buildLisp.program`: Builds an executable program out of Lisp files.

  | parameter | type         | use                           | required? |
  |-----------|--------------|-------------------------------|-----------|
  | `name`    | `string`     | Name of the program           | yes       |
  | `srcs`    | `list<path>` | List of paths to source files | yes       |
  | `deps`    | `list<drv>`  | List of dependencies          | no        |
  | `native`  | `list<drv>`  | List of native dependencies   | no        |
  | `main`    | `string`     | Entrypoint function           | no        |
  | `test`    | see "Tests"  | Specification for test suite  | no        |
  | `implementation` | see "Implementations" | Common Lisp implementation to use | no |

  The `main` parameter should be the name of a function and defaults
  to `${name}:main` (i.e. the *exported* `main` function of the
  package named after the program).

  The output of invoking this is a directory containing a
  `bin/${name}`.

* `buildLisp.bundled`: Creates a virtual dependency on a built-in library.

  Certain libraries ship with Lisp implementations, for example
  UIOP/ASDF are commonly included but many implementations also ship
  internals (such as SBCLs various `sb-*` libraries).

  This function takes a single string argument that is the name of a
  built-in library and returns a "package" that simply requires this
  library.

## Tests

Both `buildLisp.library` and `buildLisp.program` take an optional argument
`tests`, which has the following supported fields:

  | parameter    | type         | use                           | required? |
  |--------------|--------------|-------------------------------|-----------|
  | `name`       | `string`     | Name of the test suite        | no        |
  | `expression` | `string`     | Lisp expression to run tests  | yes       |
  | `srcs`       | `list<path>` | List of paths to source files | no        |
  | `native`     | `list<drv>`  | List of native dependencies   | no        |

the `expression` parameter should be a Lisp expression and will be evaluated
after loading all sources and dependencies (including library/program
dependencies). It must return a non-`NIL` value if the test suite has passed.

## Example

Using buildLisp could look like this:

```nix
{ buildLisp, lispPkgs }:

let libExample = buildLisp.library {
    name = "lib-example";
    srcs = [ ./lib.lisp ];

    deps = with lispPkgs; [
      (buildLisp.bundled "sb-posix")
      iterate
      cl-ppcre
    ];
};
in buildLisp.program {
    name = "example";
    deps = [ libExample ];
    srcs = [ ./main.lisp ];
    tests = {
      deps = [ lispPkgs.fiveam ];
      srcs = [ ./tests.lisp ];
      expression = "(fiveam:run!)";
    };
}
```

## Development REPLs

`buildLisp` builds loadable variants of both `program` and `library` derivations
(usually FASL files). Therefore it can provide a convenient way to obtain an
instance of any implementation preloaded with `buildLisp`-derivations. This
is especially useful to use as a host for Sly or SLIME.

* `buildLisp.sbcl.lispWith`, `buildLisp.ccl.lispWith`, ...:
  Creates a wrapper script preloading a Lisp implementation with various dependencies.

  This function takes a single argument which is a list of Lisp
  libraries programs or programs. The desired Lisp implementation
  will load all given derivations and all their dependencies on
  startup.

  The shortcut `buildLisp.sbclWith` for `buildLisp.sbcl.lispWith` is also provided.

* `repl` passthru attribute: `derivation.repl` is provided as a shortcut
  for `buildLisp.${implementationName}.lispWith [ derivation ]`.
  `derivation.ccl.repl`, `derivation.sbcl.repl` etc. work as well, of course
  (see also "Implementations" section).

## Implementations

Both `buildLisp.library` and `buildLisp.program` allow specifying a different
Common Lisp implementation than the default one (which is SBCL). When an
implementation is passed, `buildLisp` makes sure all dependencies are built
with that implementation as well since build artifacts from different
implementation will be incompatible with each other.

The argument taken by `implementation` is a special attribute set which
describes how to do certain tasks for a given implementation, like building
or loading a library. In case you want to use a custom implementation
description, the precise structure needed is documented in `buildLisp`'s
source code for now. `buildLisp` also exposes the following already
working implementation sets:

* `buildLisp.sbcl`: [SBCL][sbcl], our default implementation

* `buildLisp.ccl`: [CCL][ccl], similar to SBCL, but with very good macOS support

* `buildLisp.ecl`: [ECL][ecl] setup to produce statically linked binaries and
  libraries. Note that its runtime library is LGPL, so [extra conditions][lgpl-static]
  must be fulfilled when distributing binaries produced this way.

* Support for ABCL is planned.

For every of these “known” implementations, `buildLisp` will create a `passthru`
attribute named like the implementation which points to a variant of the derivation
built with said implementation. Say we have a derivation, `myDrv`, built using SBCL:
While `myDrv` and `myDrv.sbcl` are built using SBCL, `myDrv.ecl`, `myDrv.ccl` etc.
build the derivation and all its dependencies using ECL and CCL respectively.

This is useful to test portability of your derivation, but is also used internally
to speed up the “normalization” of the dependency graph. Thus it is important to
make sure that your custom implementation's name doesn't clash with one of the
“known” ones.

## Handling Implementation Specifics

When targeting multiple Common Lisp implementation, it is often necessary to
handle differing interfaces for OS interaction or to make use of special
implementation features. For this reason, `buildLisp` allows specifying
dependencies and source files for specific implementations only. This can
be utilized by having an attribute set in the list for the `deps` or `srcs`
argument: `buildLisp` will pick the value of the attribute named like the
used implementation or `default` and ignore the set completely if both
are missing.

```nix
{ buildLisp, lispPkgs }:

buildLisp.library {
  name = "mylib";

  srcs = [
    # These are included always of course
    ./package.lisp
    ./portable-lib.lisp

    # Choose right impl-* file
    {
      sbcl = ./impl-sbcl.lisp;
      ccl = ./impl-ccl.lisp;
      ecl = ./impl-ecl.lisp;
    }

    # We can also use this to inject extra files
    { ecl = ./extra-ecl-optimizations.lisp; }
  ];

  deps = [
    # Use SBCL's special bundled package, flexi-streams otherwise
    {
      sbcl = buildLisp.bundled "sb-rotate-byte";
      default = lispPkgs.flexi-streams;
    }
  ];
}
```

Additionally a `brokenOn` parameter is accepted which takes a list of
implementation names on which the derivation is not expected to work.
This only influences `meta.ci.targets` which is read by depot's CI to
check which variants (see "Implementations") of the derivation to
build, so it may not be useful outside of depot.

## Influencing the Lisp Runtime

Lisp implementations which create an executable by dumping an image
usually parse a few implementation-specific command line options on
executable startup that influence runtime settings related to things
like GC. `buildLisp` generates a wrapper which makes sure that this
never interferes with the argument parsing implemented in the actual
application, but sometimes it is useful to run an executable with
special settings. To allow this, the content of `NIX_BUILDLISP_LISP_ARGS`
is passed to the lisp implementation.

For example, you can make the underlying SBCL print its version for
any executable built with `buildLisp` (and SBCL) like this:

```console
$ env NIX_BUILDLISP_LISP_ARGS="--version" ./result/bin/🕰️
SBCL 2.1.2.nixos
```

In practice you'd probably want to specify options like
`--dynamic-space-size` or `--tls-limit` (try passing `--help` for a
full list). Naturally, these options are completely different for
different implementations.

[sbcl]: http://www.sbcl.org/
[ccl]: https://ccl.clozure.com/
[ecl]: https://common-lisp.net/project/ecl/
[lgpl-static]: https://www.gnu.org/licenses/gpl-faq.en.html#LGPLStaticVsDynamic