We need `artist_ids` in most of the queries, so let’s make them a
generated column and put an index on them.
This reduces times for getArtistNameById from ~300ms to ~20ms and for
the main “latest” table from ~250ms to ~60ms.
The `artist_has_been_snatched` logic moves into the torrent filter, so
we don’t have to work on such large records further down.
Change-Id: I5a1929bd9c81ea0031e512d01897c0e460ded077
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13240
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Or rather make it less slow lol.
I should put the artists in a computed column now.
Change-Id: I28f6546353e2930156eaeb427ba5122559aea49b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13239
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
A simple favourite system that adds an artist to your favourites when
you go to their artist page for the first time.
Also, if you fetch something from an artist, they get automatically
added to the favourites.
This is pretty much just done by adding more crap to the main SQL
query, and has the disadvantage that the `artist_has_been_snatched`
CTE slows down the query a lot for the favourite filter (by 50% or
even sometimes 150% in a weird degenerate case). I think we can
optimize this somewhat, but that needs a small rewrite of the query
CTE stages.
Change-Id: I7aa6d99dc26e24eb22ef56ffd4d2db5c6978ad48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13238
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
For the next step, we need to put more things in WITH-Clauses, so
let’s prepare it now to get a nicer diff.
Change-Id: Ibd5b67c77c87ce3d6aa0ab04ca3d66f9b90dc856
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13237
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
For latest releases, I really don’t want to see bootlegs and stuff.
Change-Id: I3a98d7b81d2ca7962dcef619d4dcfcbd18e98f84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13226
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
This is definitely faster than doing a roundtrip via a build.
Change-Id: I7a02b828462def735fdb241ce729143e90bc5c75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13236
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Let’s start improving the main page.
So far, it was just all release groups sorted by weight on a single
page, which was not super helpful (and got kinda large).
The first feature is to show the latest releases that are known.
This is done by torrent group ID. This ID does not always correspond
to the date, but can also be a very old album that gets uploaded,
or (seldomly) a group that gets merged for metadata.
We should think about restricting this to favourites, automatically
marking everything as favourite where we have an album downloaded or
clicked on the artist before, and then selectively allow to un-mark
it after the fact.
An even stronger “not interested” could be used to automatically
reclaim seedbox space once it becomes an issue.
Eventually (after implementing favourites), we should introduce a
job system that automatically updates these entries every few hours.
Maybe even have a “very interested” feature that automatically
downloads everything new for an artist?
And then a “veryvery interested” feature that also buys the thing from
bandcamp lol
Change-Id: I467c350722279ff37150f847f5014d7e0e67e626
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13225
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The RED Gazelle API is so inconsistent, smh.
Sometimes they return the release type string, sometimes the integer
representing it. So let’s handle both!
Change-Id: Ie99f01342a13269df5c1e2036aaede54d7149516
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13223
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This has been down for a while, and people have been asking me why we have a
dead link on the website.
Change-Id: I7b2bc8dd7a627678fbb8d48e86ad6eb6f1c73aa2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13222
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
For all big page reloads, we want the `<head>` of the page to start
being transmitted even while doing database requests.
So let’s use the `Wai.ResponseStream` to do exactly that. The handler
provides the contents of `<head>`, we start streaming that, meanwhile
it calculates the `<body>` and once that is ready transmits it.
This means we can load all our static resources before the page
even starts sending any body data, meaning the css and html is already
there when we reach `</html>`.
Sweet.
The `<title>` in `artistPage` was depending on the table data, so I
moved it into a separate SELECT.
We can do all of it in parallel as well. Sweet.
~~~
This also adds static file prefetching of source maps if provided.
Change-Id: Ib824430594733b4c8e86ee1096c8afba2df1a66d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13221
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Extracts the required version out of the telega.el Dockerfile (this seems to be
the authoritative source), and matches that against what nixpkgs has.
In a future commit I'll improve this to reduce the likelihood of blocking a
channel bump (by also pinning tdlib, and issuing warnings when nixpkgs has a
newer telega/tdlib).
Change-Id: I1129c1f6b38aa58eb8661f2ad9bc6fa19382d81c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13220
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Use the default emacs attribute over the versioned emacs28 attribute. On
current canon, this implies an upgrade to Emacs 29.
Emacs < 30 has some CVEs that have never been patched in those release
series. The next channel bump will upgrade pkgs.emacs to Emacs 30, so
just using the default alias seems to be the easiest solution.
See also <https://github.com/NixOS/nixpkgs/pull/386174>.
Change-Id: I1580bae138cc0801e0f3431b1a02c8a4585c2996
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13216
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Proxy the resources through our server instead of always requesting
them from the CDN, leaking lots of usage data.
This prefetches the resources at server startup (instead of e.g.
adding them to the build environment statically), which is a nice
compromise I think. It will also keep the integrity verification in
place, but that is still done in the client.
Adds `rel=preload` as well, so we start fetching asap.
No caching (yet). We could easily add a hash over the content though
so the client never has to re-request the resources.
Change-Id: I9aac80cfb1ded09e578ba2a70dcf982bf5322ff6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13215
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
The settings could actually be improved with an htmx snippet, so that
we don’t have to load the whole page every time.
Change-Id: I472940533881ff24c0111188e7265c59665bc29d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13213
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Throws out htmx for search in favor of a full page-reload for search.
This means we can reload the page and it will re-do the search.
It’s not POST anymore cause we don’t really change anything, just
do some caching when doing the search. Could probably also cache the
search result, but simple simple
Change-Id: I70b0cbac55f67c986aeecfedcf0b3968890b3b9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13212
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The `bencode` library is anything but production-grade, but it’s
enough to parse the torrent files generated by Gazelle lol.
This should help with … I haven’t really figured out yet what it helps
with I guess. But it was fun. I like the `Parse` abstraction very
much. It can parse XML and it can parse Bencode. Good.
Change-Id: If1331de423eab3e91ce9db6e2a7eb84da51b18a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13211
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Experiment: can we convert any Html-producing conduit into a
self-updating htmx snippet? Yes!
Add the resulting handler and initial snippet, and it will poll the
handler every second or so until the conduit stops emitting values.
Change-Id: I0bed17a5db8691a19df6c334ae1268dd6a3d3123
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13209
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The unpacked lrclib dump is 40GB at this point, but compresses very
well (to about 10GB give or take). So I thought about how to store it
in a compressed on disk, but still allow sqlite to query
efficiently (i.e. use a compression format that has arbitrary seek
indices).
Squashfs! with zstd compression!
```
mksquashfs ~/.cache/lyric-bak/ \
~/tmp/squashtest \
-comp zstd \
-Xcompression-level 5
```
Still need to write a helper that does the download & compression for
me to update the database once in a blue moon.
Unfortunately, sqlite-utils does not allow URL connection strings yet,
so I had to do a horrible SQL interpolation bit.
Issue tracker: https://mastodon.xyz/@Profpatsch/114083481108799334
This also changes the query to filter out every track that has neither
plain nor synched lyrics (yeah those exist for some reason?).
Change-Id: Ief0519ae8d9e5f14b66f6df6e3bcecae911ea142
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13208
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This error type is a bit less annoying to use in other errors, as it
doesn't hang onto bytes from the input.
Change-Id: I4b969717d443f5e04dd2086baeb946a98c4f9d7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13206
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
This makes it possible for external consumers to deal with this error.
Change-Id: I70b16efb99c6a98c75b1672ed0618ac963f5f79d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13194
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
This adds code allowing to parse (and emit) the "internal-json" log
format that Nixcpp emits.
Move the Verbosity struct from worker_protocol.rs out into
src/log/mod.rs, it's not only used there.
Change-Id: Ia35e739c83e06080a677b576616bdd0d4a821cf4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13167
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: edef <edef@edef.eu>
See 72dd5818b7/CHANGELOG.md
for the nom changelog.
Most notably, there's now a .parse() to be added:
`combinator(arg)(input)` -> `combinator(arg).parse(input)`
There also doesn't need to be a tuple combinator (it's implemented on
tuples directly).
This also refactors the string / byte field parsing parts, to make them
more concise.
Change-Id: I9e8a3cedd07d6705be391898eb6a486fb8164069
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13193
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: Brian Olsen <me@griff.name>
Update the tvix cli's -I option so that it aligns more closely with
nix's behavior: prepending entries to the list of lookup paths provided
by the NIX_PATH environment variable. Before this commit, using the -I
option would instead override and ignore the NIX_PATH variable.
Additionally, update the option's long name and help text to match the
new behavior.
While the tvix cli's interface does not appear to be attempting to mimic
nix exactly, I think this particular case of the -I option's diverging
behavior will inevitably surprise users because it's name, presumably
short for "include" and being similar to gcc's flag, evokes additivity.
The prior implementation hinted at this difference with the help text
and the long name, --nix-search-path, but I still suspect users will be
confused on first usage (at least I was). If we're willing to pay the
maintenance costs of additional code, we can avoid this and provide a
slightly smoother user experience.
Changes were tested by buiding the tvix cli, adding it to the PATH, and
executing simple tests as in the following bash script
mg build //tvix/cli
PATH="$PWD/result/bin:$PATH"
one=$(mktemp) && echo "=> $one :: path" > "$one"
two=$(mktemp) && echo "=> $two :: path" > "$two"
dir1=$(mktemp -d) && file1="$dir1/file1" && echo "=> $file1 :: path" > "$file1"
dir2=$(mktemp -d) && file2="$dir2/file2" && echo "=> $file2 :: path" > "$file2"
# NIX_PATH works with a single non-prefixed lookup path.
NIX_PATH="$dir1" tvix -E "<file1>" | cmp - "$file1"
# NIX_PATH works with multiple non-prefixed lookup paths.
NIX_PATH="$dir1:$dir2" tvix -E "<file2>" | cmp - "$file2"
# NIX_PATH works with a single prefixed lookup path.
NIX_PATH="one=$one" tvix -E "<one>" | cmp - "$one"
# NIX_PATH works with multiple prefixed lookup paths.
NIX_PATH="one=$one:two=$two" tvix -E "<one>" | cmp - "$one"
NIX_PATH="one=$one:two=$two" tvix -E "<two>" | cmp - "$two"
# NIX_PATH first entry takes precedence.
NIX_PATH="one=$one:one=$two" tvix -E "<one>" | cmp - "$one"
# The -I option works with a single non-prefixed lookup path.
tvix -I "$dir1" -E "<file1>" | cmp - "$file1"
# The -I option works with multiple non-prefixed lookup paths.
tvix -I "$dir1" -I "$dir2" -E "<file2>" | cmp - "$file2"
# The -I option works with a single prefixed lookup path.
tvix -I "one=$one" -E "<one>" | cmp - "$one"
# The --extra-nix-path option works with a single prefixed lookup path.
tvix --extra-nix-path "one=$one" -E "<one>" | cmp - "$one"
# The -I options works when passed multiple times with prefixed lookup paths.
tvix -I "one=$one" -I "two=$two" -E "<one>" | cmp - "$one"
tvix -I "one=$one" -I "two=$two" -E "<two>" | cmp - "$two"
# The first -I option takes precedence.
tvix -I "one=$one" -I "one=$two" -E "<one>" | cmp - "$one"
# Both NIX_PATH and the -I option work together and are additive.
NIX_PATH="one=$one" tvix -I "two=$two" -E "<one>" | cmp - "$one"
NIX_PATH="one=$one" tvix -I "two=$two" -E "<two>" | cmp - "$two"
# The -I option takes precedence over NIX_PATH.
NIX_PATH="one=$one" tvix -I "one=$two" -E "<one>" | cmp - "$two"
rm "$one"
rm "$two"
rm "$file1" && rmdir "$dir1"
rm "$file2" && rmdir "$dir2"
The above script assumes it's being run from inside the depot.
Change-Id: I153e6de57939c0eeca1f9e479d807862ab69b2de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13189
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Buildkite introduced a weird check that determines whether the bootstrap agent
path (?) matches the binary that the agent was started from:
https://github.com/buildkite/agent/pull/3123
They did this to work around some internal development flow problems. However,
this check is toggled by whether or not the `buildNumber` compile-time variable
is set to the special `x` sentinel value.
In their publicly released binaries (which we do not use, of course), this is
set to some other value. In Nix builds they are at the default sentinel value,
causing crashes at startup because of the wrapper script not matching the binary
path:
```
buildkite-agent: fatal: check binary paths: mismatched buildkite-agent paths: host="/nix/store/rmp9g00bppi8yimr0ngnx6490w196in8-buildkite-agent-3.89.0/bin/.buildkite-agent-wrapped" bootstrap="/nix/store/rmp9g00bppi8yimr0ngnx6490w196in8-buildkite-agent-3.89.0/bin/buildkite-agent"
```
To work around this we just set the build "number" to `nix`.
Change-Id: I794861aeaf63764689148cae841ce56f88752186
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13205
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
I think it's nice to have a place to keep around things that are
relevant, but not necessarily accurate anymore. We could also expand
this to actually describe the history of depot, but I lack the context
for that as I haven't been around from the beginning.
We may want to redirect the link from README to the monorepo doc here.
Change-Id: I971f4ab873a66ba66719a0795d86b4de0e32d32b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13039
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Our maintenance branch includes additional concurrency fixes which are not in
the upstream 2.3 branch.
These issues are fixed in C++ Nix HEAD, but in a more invasive way (by removing
the second set of locks completely).
This also retains additional debug information in the built binaries to make
future issues easier to debug.
Change-Id: I4e7a8baabd059c96404822d9634df52c403a869f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13135
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Fix "environent" typo in the Builder Protocol document. Remove "todo"
label from the nixVersion builtin because it appears to have been
implemented in 7fe1331570
Change-Id: I64aaa7cc93c8c6978cedd3bfacf756dacb4c7fc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13188
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Optimisations seem to cause *oauth2-redirect-uri* to be baked in if we
try to check the environment in DEFVAR.
Change-Id: Id53b880061239141936298f673ff8e745442af94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13186
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
* //tvix/*-go: regenerate protobuf files
* //tvix/boot:
- Explicitly set compression method of mkBinaryCache which has
made this configurable and (at the same time) changed the default.
- Adjust to change of extension of the nar files from .nar.xz to
plain .xz.
Change-Id: Ie79ea8e0ac8fe04ae01f5558bffca93e9314f56d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13174
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
