maurice/snix
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(web): import store configuration/composition document

Browse source 
Put it in Guides, as it provides some examples at the end as well.

Change-Id: Ic5cd78bcda09c3bb82eeaa88ff0c959c4c876bd7
Reviewed-on: https://cl.snix.dev/c/snix/+/30445
Autosubmit: Florian Klink <flokli@flokli.de>
Reviewed-by: Yureka <snix@yuka.dev>
Tested-by: besadii
This commit is contained in:
Florian Klink 2025-05-04 18:04:10 +03:00 • committed by clbot
parent 7b329b402c
commit dd3b447428
1 changed files with 38 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

64
snix/docs/src/castore/store-configuration.md → web/content/docs/guides/store-configuration-composition.md
View file

@ -1,21 +1,33 @@
# Store Configuration ---
title: "Store Configuration / Composition"
slug: store-configuration-composition
description: ""
summary: ""
date: 2025-05-03T16:00:33+00:00
lastmod: 2025-05-03T16:00:33+00:00
draft: false
weight: 15
toc: true
---
Currently, tvix-store (and tvix-cli) expose three different `--*-service-addr` Currently, snix-store, snix-cli and other CLI endpoints expose three different
CLI args, describing how to talk to the three different stores. `--*-service-addr` CLI args, describing how to talk to the three different
stores.
Depending on the CLI entrypoint, they have different defaults: Depending on the CLI entrypoint, they have different defaults:
- `tvix-cli` defaults to in-memory variants (`ServiceUrlsMemory`). - `snix-cli` defaults to in-memory variants (`ServiceUrlsMemory`).
- `tvix-store daemon` defaults to using a local filesystem-based backend for - `snix-store daemon` defaults to using a local filesystem-based backend for
blobs, and redb backends for `DirectoryService` and `PathInfoService` blobs, and redb backends for `DirectoryService` and `PathInfoService`
(`ServiceUrls`). (`ServiceUrls`).
- other `tvix-store` entrypoints, as well as `nar-bridge` default to talking to - other `snix-store` entrypoints, as well as `nar-bridge` default to talking to
a `tvix-store` gRPC daemon (`ServiceUrlsGrpc`). a `snix-store` gRPC daemon (`ServiceUrlsGrpc`).
The exact config and paths can be inspected by invoking `--help` on each of The exact config and paths can be inspected by invoking `--help` on each of
these entrypoints, and it's of course possible to change this config, for these entrypoints, and it's of course possible to change this config, for
example in case everything should be done from a single binary, without a daemon example in case everything should be done from a single binary, without a daemon
in between. in between.
There currently is no caching on the client side wired up yet, and some (known) There currently is no caching on the client side wired up yet, and some (known)
unnecessary roundtrips (which can be removed after some refactoring), so for unnecessary roundtrips (which can be removed after some refactoring), so for
everything except testing purposes you might want to directly connect to the everything except testing purposes you might want to directly connect to the
@ -23,13 +35,13 @@ data stores, or use Store Composition to have caching, (and describe more
complicated fetch-through configs). complicated fetch-through configs).
## Store Composition ## Store Composition
Internally, `tvix-castore` supports composing multiple instances of `BlobService`, Internally, `snix-[ca]store` supports composing multiple instances of
`DirectoryService` (and `PathInfoService`) together. `BlobService`, `DirectoryService` (and `PathInfoService`) together.
It allows describing more complicated "hierarchies"/"tiers" of different This allows describing more complicated "hierarchies"/"tiers" of different
service types. It supports combining different storage backend/substituters/ service types, by combining different storage backend/substituters/
caches, and combining them in a DAG of some sort, ultimately exposing the same caches in a DAG of some sort, with the root still exposing the same (trait)
(trait) interface as a single store. interface as a single store.
The three individual URLs exposed in the CLI currently are internally converted The three individual URLs exposed in the CLI currently are internally converted
to a composition with just one instance of each store (at the "root" name). to a composition with just one instance of each store (at the "root" name).
@ -50,10 +62,10 @@ only.
### CLI usage ### CLI usage
However, if you're okay with these caveats, and want to configure some caching However, if you're okay with these caveats, and want to configure some caching
today, using the existing CLI entrypoints, you can enable the today, using the existing CLI entrypoints, you can enable the
`xp-composition-cli` feature flag in the `tvix-store` crate. `xp-composition-cli` feature flag in the `snix-store` crate.
With `cargo`, this can be enabled by passing With `cargo`, this can be enabled by passing
`--features tvix-store/xp-composition-cli` to a `cargo build` / `cargo run` `--features snix-store/xp-composition-cli` to a `cargo build` / `cargo run`
invocation. [^1] invocation. [^1]
If enabled, CLI entrypoints get a `--experimental-store-composition` arg, which If enabled, CLI entrypoints get a `--experimental-store-composition` arg, which
@ -65,8 +77,10 @@ attribute, (`DirectoryService`s in `directoryservices`, and `PathInfoService`s
in `pathinfoservices` respectively), and requires one named "root". in `pathinfoservices` respectively), and requires one named "root".
### Library usage ### Library usage
The store composition code can be accessed via `snix_castore::composition`, and The store composition code and documentation and examples can be viewed at
`snix_store::composition`. [`snix_castore::composition`][rustdoc-castore-composition] and
[`snix_store::composition`][rustdoc-store-composition].
Make sure to check them out.
A global "registry" can be used to make other (out-of-tree) "types" of stores A global "registry" can be used to make other (out-of-tree) "types" of stores
known to the composition machinery. known to the composition machinery.
@ -74,9 +88,6 @@ known to the composition machinery.
In terms of config format, you're also not required to use TOML, but anything In terms of config format, you're also not required to use TOML, but anything
`serde` can deserialize. `serde` can deserialize.
Make sure to check the module-level docstrings and code examples for
`snix_castore::composition`.
### Composition config format ### Composition config format
Below examples are in the format accepted by the CLI, using the Below examples are in the format accepted by the CLI, using the
`blobservices` / `directoryservices` / `pathinfoservices` namespace/attribute to `blobservices` / `directoryservices` / `pathinfoservices` namespace/attribute to
@ -100,7 +111,7 @@ far = "far"
[blobservices.near] [blobservices.near]
type = "objectstore" type = "objectstore"
object_store_url = "file:///tmp/tvix/blobservice" object_store_url = "file:///tmp/snix/blobservice"
object_store_options = {} object_store_options = {}
[blobservices.far] [blobservices.far]
@ -129,7 +140,7 @@ url = "grpc+http://localhost:8000"
# […] blobservices/directoryservices go here […] # […] blobservices/directoryservices go here […]
``` ```
### Example: Self-contained fetch-through tvix-store for `cache.nixos.org`. ### Example: Self-contained fetch-through snIx-store for `cache.nixos.org`.
This provides a `PathInfoService` "containing" `PathInfo` that are in This provides a `PathInfoService` "containing" `PathInfo` that are in
`cache.nixos.org`. `cache.nixos.org`.
@ -146,13 +157,13 @@ services, directory services).
``` ```
[blobservices.root] [blobservices.root]
type = "objectstore" type = "objectstore"
object_store_url = "file:///var/lib/tvix-store/blobs.object_store" object_store_url = "file:///var/lib/snix-store/blobs.object_store"
object_store_options = {} object_store_options = {}
[directoryservices.root] [directoryservices.root]
type = "redb" type = "redb"
is_temporary = false is_temporary = false
path = "/var/lib/tvix-store/directories.redb" path = "/var/lib/snix-store/directories.redb"
[pathinfoservices.root] [pathinfoservices.root]
type = "cache" type = "cache"
@ -162,7 +173,7 @@ far = "cache-nixos-org"
[pathinfoservices.redb] [pathinfoservices.redb]
type = "redb" type = "redb"
is_temporary = false is_temporary = false
path = "/var/lib/tvix-store/pathinfo.redb" path = "/var/lib/snix-store/pathinfo.redb"
[pathinfoservices.cache-nixos-org] [pathinfoservices.cache-nixos-org]
type = "nix" type = "nix"
@ -172,5 +183,6 @@ blob_service = "root"
directory_service = "root" directory_service = "root"
``` ```
[^1]: In some leaf binary crates, this can also be controlled via the `xp-store-composition-cli` feature in the leaf crate itself. [^1]: In some leaf binary crates, this can also be controlled via the `xp-store-composition-cli` feature in the leaf crate itself.
[rustdoc-castore-composition]: https://snix.dev/rustdoc/snix_castore/composition/index.html
[rustdoc-store-composition]: https://snix.dev/rustdoc/snix_store/composition/index.html