Let’s start improving the main page.
So far, it was just all release groups sorted by weight on a single
page, which was not super helpful (and got kinda large).
The first feature is to show the latest releases that are known.
This is done by torrent group ID. This ID does not always correspond
to the date, but can also be a very old album that gets uploaded,
or (seldomly) a group that gets merged for metadata.
We should think about restricting this to favourites, automatically
marking everything as favourite where we have an album downloaded or
clicked on the artist before, and then selectively allow to un-mark
it after the fact.
An even stronger “not interested” could be used to automatically
reclaim seedbox space once it becomes an issue.
Eventually (after implementing favourites), we should introduce a
job system that automatically updates these entries every few hours.
Maybe even have a “very interested” feature that automatically
downloads everything new for an artist?
And then a “veryvery interested” feature that also buys the thing from
bandcamp lol
Change-Id: I467c350722279ff37150f847f5014d7e0e67e626
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13225
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The RED Gazelle API is so inconsistent, smh.
Sometimes they return the release type string, sometimes the integer
representing it. So let’s handle both!
Change-Id: Ie99f01342a13269df5c1e2036aaede54d7149516
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13223
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This has been down for a while, and people have been asking me why we have a
dead link on the website.
Change-Id: I7b2bc8dd7a627678fbb8d48e86ad6eb6f1c73aa2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13222
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
For all big page reloads, we want the `<head>` of the page to start
being transmitted even while doing database requests.
So let’s use the `Wai.ResponseStream` to do exactly that. The handler
provides the contents of `<head>`, we start streaming that, meanwhile
it calculates the `<body>` and once that is ready transmits it.
This means we can load all our static resources before the page
even starts sending any body data, meaning the css and html is already
there when we reach `</html>`.
Sweet.
The `<title>` in `artistPage` was depending on the table data, so I
moved it into a separate SELECT.
We can do all of it in parallel as well. Sweet.
~~~
This also adds static file prefetching of source maps if provided.
Change-Id: Ib824430594733b4c8e86ee1096c8afba2df1a66d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13221
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Extracts the required version out of the telega.el Dockerfile (this seems to be
the authoritative source), and matches that against what nixpkgs has.
In a future commit I'll improve this to reduce the likelihood of blocking a
channel bump (by also pinning tdlib, and issuing warnings when nixpkgs has a
newer telega/tdlib).
Change-Id: I1129c1f6b38aa58eb8661f2ad9bc6fa19382d81c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13220
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Use the default emacs attribute over the versioned emacs28 attribute. On
current canon, this implies an upgrade to Emacs 29.
Emacs < 30 has some CVEs that have never been patched in those release
series. The next channel bump will upgrade pkgs.emacs to Emacs 30, so
just using the default alias seems to be the easiest solution.
See also <https://github.com/NixOS/nixpkgs/pull/386174>.
Change-Id: I1580bae138cc0801e0f3431b1a02c8a4585c2996
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13216
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Proxy the resources through our server instead of always requesting
them from the CDN, leaking lots of usage data.
This prefetches the resources at server startup (instead of e.g.
adding them to the build environment statically), which is a nice
compromise I think. It will also keep the integrity verification in
place, but that is still done in the client.
Adds `rel=preload` as well, so we start fetching asap.
No caching (yet). We could easily add a hash over the content though
so the client never has to re-request the resources.
Change-Id: I9aac80cfb1ded09e578ba2a70dcf982bf5322ff6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13215
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
The settings could actually be improved with an htmx snippet, so that
we don’t have to load the whole page every time.
Change-Id: I472940533881ff24c0111188e7265c59665bc29d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13213
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Throws out htmx for search in favor of a full page-reload for search.
This means we can reload the page and it will re-do the search.
It’s not POST anymore cause we don’t really change anything, just
do some caching when doing the search. Could probably also cache the
search result, but simple simple
Change-Id: I70b0cbac55f67c986aeecfedcf0b3968890b3b9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13212
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The `bencode` library is anything but production-grade, but it’s
enough to parse the torrent files generated by Gazelle lol.
This should help with … I haven’t really figured out yet what it helps
with I guess. But it was fun. I like the `Parse` abstraction very
much. It can parse XML and it can parse Bencode. Good.
Change-Id: If1331de423eab3e91ce9db6e2a7eb84da51b18a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13211
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Experiment: can we convert any Html-producing conduit into a
self-updating htmx snippet? Yes!
Add the resulting handler and initial snippet, and it will poll the
handler every second or so until the conduit stops emitting values.
Change-Id: I0bed17a5db8691a19df6c334ae1268dd6a3d3123
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13209
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The unpacked lrclib dump is 40GB at this point, but compresses very
well (to about 10GB give or take). So I thought about how to store it
in a compressed on disk, but still allow sqlite to query
efficiently (i.e. use a compression format that has arbitrary seek
indices).
Squashfs! with zstd compression!
```
mksquashfs ~/.cache/lyric-bak/ \
~/tmp/squashtest \
-comp zstd \
-Xcompression-level 5
```
Still need to write a helper that does the download & compression for
me to update the database once in a blue moon.
Unfortunately, sqlite-utils does not allow URL connection strings yet,
so I had to do a horrible SQL interpolation bit.
Issue tracker: https://mastodon.xyz/@Profpatsch/114083481108799334
This also changes the query to filter out every track that has neither
plain nor synched lyrics (yeah those exist for some reason?).
Change-Id: Ief0519ae8d9e5f14b66f6df6e3bcecae911ea142
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13208
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This error type is a bit less annoying to use in other errors, as it
doesn't hang onto bytes from the input.
Change-Id: I4b969717d443f5e04dd2086baeb946a98c4f9d7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13206
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
This makes it possible for external consumers to deal with this error.
Change-Id: I70b16efb99c6a98c75b1672ed0618ac963f5f79d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13194
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
This adds code allowing to parse (and emit) the "internal-json" log
format that Nixcpp emits.
Move the Verbosity struct from worker_protocol.rs out into
src/log/mod.rs, it's not only used there.
Change-Id: Ia35e739c83e06080a677b576616bdd0d4a821cf4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13167
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: edef <edef@edef.eu>
See 72dd5818b7/CHANGELOG.md
for the nom changelog.
Most notably, there's now a .parse() to be added:
`combinator(arg)(input)` -> `combinator(arg).parse(input)`
There also doesn't need to be a tuple combinator (it's implemented on
tuples directly).
This also refactors the string / byte field parsing parts, to make them
more concise.
Change-Id: I9e8a3cedd07d6705be391898eb6a486fb8164069
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13193
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: Brian Olsen <me@griff.name>
Update the tvix cli's -I option so that it aligns more closely with
nix's behavior: prepending entries to the list of lookup paths provided
by the NIX_PATH environment variable. Before this commit, using the -I
option would instead override and ignore the NIX_PATH variable.
Additionally, update the option's long name and help text to match the
new behavior.
While the tvix cli's interface does not appear to be attempting to mimic
nix exactly, I think this particular case of the -I option's diverging
behavior will inevitably surprise users because it's name, presumably
short for "include" and being similar to gcc's flag, evokes additivity.
The prior implementation hinted at this difference with the help text
and the long name, --nix-search-path, but I still suspect users will be
confused on first usage (at least I was). If we're willing to pay the
maintenance costs of additional code, we can avoid this and provide a
slightly smoother user experience.
Changes were tested by buiding the tvix cli, adding it to the PATH, and
executing simple tests as in the following bash script
mg build //tvix/cli
PATH="$PWD/result/bin:$PATH"
one=$(mktemp) && echo "=> $one :: path" > "$one"
two=$(mktemp) && echo "=> $two :: path" > "$two"
dir1=$(mktemp -d) && file1="$dir1/file1" && echo "=> $file1 :: path" > "$file1"
dir2=$(mktemp -d) && file2="$dir2/file2" && echo "=> $file2 :: path" > "$file2"
# NIX_PATH works with a single non-prefixed lookup path.
NIX_PATH="$dir1" tvix -E "<file1>" | cmp - "$file1"
# NIX_PATH works with multiple non-prefixed lookup paths.
NIX_PATH="$dir1:$dir2" tvix -E "<file2>" | cmp - "$file2"
# NIX_PATH works with a single prefixed lookup path.
NIX_PATH="one=$one" tvix -E "<one>" | cmp - "$one"
# NIX_PATH works with multiple prefixed lookup paths.
NIX_PATH="one=$one:two=$two" tvix -E "<one>" | cmp - "$one"
NIX_PATH="one=$one:two=$two" tvix -E "<two>" | cmp - "$two"
# NIX_PATH first entry takes precedence.
NIX_PATH="one=$one:one=$two" tvix -E "<one>" | cmp - "$one"
# The -I option works with a single non-prefixed lookup path.
tvix -I "$dir1" -E "<file1>" | cmp - "$file1"
# The -I option works with multiple non-prefixed lookup paths.
tvix -I "$dir1" -I "$dir2" -E "<file2>" | cmp - "$file2"
# The -I option works with a single prefixed lookup path.
tvix -I "one=$one" -E "<one>" | cmp - "$one"
# The --extra-nix-path option works with a single prefixed lookup path.
tvix --extra-nix-path "one=$one" -E "<one>" | cmp - "$one"
# The -I options works when passed multiple times with prefixed lookup paths.
tvix -I "one=$one" -I "two=$two" -E "<one>" | cmp - "$one"
tvix -I "one=$one" -I "two=$two" -E "<two>" | cmp - "$two"
# The first -I option takes precedence.
tvix -I "one=$one" -I "one=$two" -E "<one>" | cmp - "$one"
# Both NIX_PATH and the -I option work together and are additive.
NIX_PATH="one=$one" tvix -I "two=$two" -E "<one>" | cmp - "$one"
NIX_PATH="one=$one" tvix -I "two=$two" -E "<two>" | cmp - "$two"
# The -I option takes precedence over NIX_PATH.
NIX_PATH="one=$one" tvix -I "one=$two" -E "<one>" | cmp - "$two"
rm "$one"
rm "$two"
rm "$file1" && rmdir "$dir1"
rm "$file2" && rmdir "$dir2"
The above script assumes it's being run from inside the depot.
Change-Id: I153e6de57939c0eeca1f9e479d807862ab69b2de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13189
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Buildkite introduced a weird check that determines whether the bootstrap agent
path (?) matches the binary that the agent was started from:
https://github.com/buildkite/agent/pull/3123
They did this to work around some internal development flow problems. However,
this check is toggled by whether or not the `buildNumber` compile-time variable
is set to the special `x` sentinel value.
In their publicly released binaries (which we do not use, of course), this is
set to some other value. In Nix builds they are at the default sentinel value,
causing crashes at startup because of the wrapper script not matching the binary
path:
```
buildkite-agent: fatal: check binary paths: mismatched buildkite-agent paths: host="/nix/store/rmp9g00bppi8yimr0ngnx6490w196in8-buildkite-agent-3.89.0/bin/.buildkite-agent-wrapped" bootstrap="/nix/store/rmp9g00bppi8yimr0ngnx6490w196in8-buildkite-agent-3.89.0/bin/buildkite-agent"
```
To work around this we just set the build "number" to `nix`.
Change-Id: I794861aeaf63764689148cae841ce56f88752186
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13205
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
I think it's nice to have a place to keep around things that are
relevant, but not necessarily accurate anymore. We could also expand
this to actually describe the history of depot, but I lack the context
for that as I haven't been around from the beginning.
We may want to redirect the link from README to the monorepo doc here.
Change-Id: I971f4ab873a66ba66719a0795d86b4de0e32d32b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13039
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Our maintenance branch includes additional concurrency fixes which are not in
the upstream 2.3 branch.
These issues are fixed in C++ Nix HEAD, but in a more invasive way (by removing
the second set of locks completely).
This also retains additional debug information in the built binaries to make
future issues easier to debug.
Change-Id: I4e7a8baabd059c96404822d9634df52c403a869f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13135
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Fix "environent" typo in the Builder Protocol document. Remove "todo"
label from the nixVersion builtin because it appears to have been
implemented in 7fe1331570
Change-Id: I64aaa7cc93c8c6978cedd3bfacf756dacb4c7fc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13188
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Optimisations seem to cause *oauth2-redirect-uri* to be baked in if we
try to check the environment in DEFVAR.
Change-Id: Id53b880061239141936298f673ff8e745442af94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13186
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
* //tvix/*-go: regenerate protobuf files
* //tvix/boot:
- Explicitly set compression method of mkBinaryCache which has
made this configurable and (at the same time) changed the default.
- Adjust to change of extension of the nar files from .nar.xz to
plain .xz.
Change-Id: Ie79ea8e0ac8fe04ae01f5558bffca93e9314f56d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13174
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Build //3p/lisp from pkgs proper, i.e. nixpkgs' nixos-unstable channel
instead of nixos-23.11 (yikes).
Basically, multiple package sets are attached to the different lisp
implementations now instead of having a “generic” lispPackages
set (which defaults to sbcl). We can just use that instead even though
it looks a bit weird having `srcOnly sbcl.pkgs.foo` everywhere when the
packages is not necessarily related to SBCL.
We could in theory create a source only package set by abusing how the
infrastructure works internally, but it's probably somewhat brittle:
callPackage (pkgs.path + "/pkgs/development/lisp-modules/imported.nix") {
build-asdf-system = { src, ... }: src;
}
Since we do a pretty hefty jump in package versions, many packages have
to be adapted to internal changes and restructuring:
- bordeaux-threads
- cffi
- cl-colors2 (which has been deprecated, but is still required by other
packages)
- cl-smtp
- cl-plus-ssl
- cl-prevalence
- hunchentoot (compiling the asd file no longer seemed to work)
- ironclad (fixes for SBCL compiler warnings caused a CCL compiler
warning)
- nibbles (revert the only commit to sbcl-opt/x86-vm.lisp that's new
compared to canon since it broke compilation for unknown reasons)
The following new packages had to be added as existing packages added
new dependencies:
- frugal-uuid, frugal-uuid/non-frugal
- trivial-clock
Change-Id: I8b94894df0357907cf2b27cf1e34a7e804b68e02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13134
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I am already using this outside of depot where it isn't under VCS. It
makes sense to canonicalize it, also to ensure that it stays somewhat
generic.
Change-Id: I5595d98ab4198794c395feb4d3c08df1e2d01a36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13184
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Remember //nix/escapeExecline? Well, this is worse (for some possible
meanings of the word). Instead of taking a list and escaping/rendering
it to an execline script, Execline reimplements part of execlineb(1):
It takes a (nested) list where any enclosed list signifies an execline
block and produces the corresponding execline argv form as described in
execline-block(7). This means the result of Execline can directly be
executed using •SH without the need for execlineb(1).
Consequently, execlineb(1)'s management of positional parameters and the
environment are not available. This is fine for the intended purpose of
Execline (glueing together shell commands efficiently without messing
around with the FFI in BQN for pipe(2) etc.).
Change-Id: Ief69b1bab919c16b6e39c3f5dc3db628766c5a8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13180
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
These days highlight.js in mdbook includes syntax highlighting for Nix
out of the box, there's no more need to ship our own variant of it.
Change-Id: Idd453c2efe97707ae861de9fa08c9dab71ea7876
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13178
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: adisbladis <adisbladis@gmail.com>
Tested-by: BuildkiteCI
Currently ingest_entries panics. This makes it hard to use
ingest_entries in parallel with other processes. When another process
runs into an error and wants to return so, while ingest_entries has
already started and has no entries yet, we're forced to panic.
With this change ingest_entries will return an error when there are no
entries, thus allowing the user to handle the errors as they please.
Change-Id: I78b85bf18f52af8c157d6bedad6019fd4398250a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13146
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Bob van der Linden <bobvanderlinden@gmail.com>
Reviewed-by: Bob van der Linden <bobvanderlinden@gmail.com>
Tested-by: BuildkiteCI
Something recently caused us to replace Docker with Podman (I guess a default
changed in nixpkgs? I don't remember making the change explicitly), which broke
the reindexing unit.
Change-Id: I1d3453ed970e536abb540c6ef79765cfda271810
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13173
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
