Some experiments in speeding up search efficiency.
This is more of a “is this possible” thing, and it looks like it’s
possible, but does not really improve anything regarding the rate
limiting.
The idea is that we can start everything at the same time as async,
but use a semaphore to have only 5 things run at once. That also means
that as soon as something is done, we immediately start the next task.
The asyncs are guaranteed to be cleaned up by the `ResourceT`
wrapper (eventually).
It’s pretty cool how Conduit makes writing these side-effecting things
pretty possible.
Change-Id: Ibadead7db293373b415840960602fa71920fc653
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13246
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This bunches up 5 search page requests to run at the same time.
We use a conduit now, so we could get smart about returning partial
results and such (if for example upstream puts us into the rate limit,
which they do after 10 requests.
Change-Id: Idbb174334fa499c16b3426a8d129deaf3a1d3b0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13245
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
I missed this! It speeds up `/artist` by another 50%.
Change-Id: I138c38185ea7fb606a139b436b9c956f7b8e1a37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13244
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
move the label stuff into its own temporary module (until we figure
out what to put into pa-label).
Also rewrite a few things to use t2/t3.
Change-Id: I8cc8678ec01a56d6c738eb4833a3ba566a7a1e20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13242
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
We need `artist_ids` in most of the queries, so let’s make them a
generated column and put an index on them.
This reduces times for getArtistNameById from ~300ms to ~20ms and for
the main “latest” table from ~250ms to ~60ms.
The `artist_has_been_snatched` logic moves into the torrent filter, so
we don’t have to work on such large records further down.
Change-Id: I5a1929bd9c81ea0031e512d01897c0e460ded077
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13240
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Or rather make it less slow lol.
I should put the artists in a computed column now.
Change-Id: I28f6546353e2930156eaeb427ba5122559aea49b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13239
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
A simple favourite system that adds an artist to your favourites when
you go to their artist page for the first time.
Also, if you fetch something from an artist, they get automatically
added to the favourites.
This is pretty much just done by adding more crap to the main SQL
query, and has the disadvantage that the `artist_has_been_snatched`
CTE slows down the query a lot for the favourite filter (by 50% or
even sometimes 150% in a weird degenerate case). I think we can
optimize this somewhat, but that needs a small rewrite of the query
CTE stages.
Change-Id: I7aa6d99dc26e24eb22ef56ffd4d2db5c6978ad48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13238
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
For the next step, we need to put more things in WITH-Clauses, so
let’s prepare it now to get a nicer diff.
Change-Id: Ibd5b67c77c87ce3d6aa0ab04ca3d66f9b90dc856
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13237
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
For latest releases, I really don’t want to see bootlegs and stuff.
Change-Id: I3a98d7b81d2ca7962dcef619d4dcfcbd18e98f84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13226
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Let’s start improving the main page.
So far, it was just all release groups sorted by weight on a single
page, which was not super helpful (and got kinda large).
The first feature is to show the latest releases that are known.
This is done by torrent group ID. This ID does not always correspond
to the date, but can also be a very old album that gets uploaded,
or (seldomly) a group that gets merged for metadata.
We should think about restricting this to favourites, automatically
marking everything as favourite where we have an album downloaded or
clicked on the artist before, and then selectively allow to un-mark
it after the fact.
An even stronger “not interested” could be used to automatically
reclaim seedbox space once it becomes an issue.
Eventually (after implementing favourites), we should introduce a
job system that automatically updates these entries every few hours.
Maybe even have a “very interested” feature that automatically
downloads everything new for an artist?
And then a “veryvery interested” feature that also buys the thing from
bandcamp lol
Change-Id: I467c350722279ff37150f847f5014d7e0e67e626
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13225
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The RED Gazelle API is so inconsistent, smh.
Sometimes they return the release type string, sometimes the integer
representing it. So let’s handle both!
Change-Id: Ie99f01342a13269df5c1e2036aaede54d7149516
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13223
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
For all big page reloads, we want the `<head>` of the page to start
being transmitted even while doing database requests.
So let’s use the `Wai.ResponseStream` to do exactly that. The handler
provides the contents of `<head>`, we start streaming that, meanwhile
it calculates the `<body>` and once that is ready transmits it.
This means we can load all our static resources before the page
even starts sending any body data, meaning the css and html is already
there when we reach `</html>`.
Sweet.
The `<title>` in `artistPage` was depending on the table data, so I
moved it into a separate SELECT.
We can do all of it in parallel as well. Sweet.
~~~
This also adds static file prefetching of source maps if provided.
Change-Id: Ib824430594733b4c8e86ee1096c8afba2df1a66d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13221
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Proxy the resources through our server instead of always requesting
them from the CDN, leaking lots of usage data.
This prefetches the resources at server startup (instead of e.g.
adding them to the build environment statically), which is a nice
compromise I think. It will also keep the integrity verification in
place, but that is still done in the client.
Adds `rel=preload` as well, so we start fetching asap.
No caching (yet). We could easily add a hash over the content though
so the client never has to re-request the resources.
Change-Id: I9aac80cfb1ded09e578ba2a70dcf982bf5322ff6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13215
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
The settings could actually be improved with an htmx snippet, so that
we don’t have to load the whole page every time.
Change-Id: I472940533881ff24c0111188e7265c59665bc29d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13213
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Throws out htmx for search in favor of a full page-reload for search.
This means we can reload the page and it will re-do the search.
It’s not POST anymore cause we don’t really change anything, just
do some caching when doing the search. Could probably also cache the
search result, but simple simple
Change-Id: I70b0cbac55f67c986aeecfedcf0b3968890b3b9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13212
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The `bencode` library is anything but production-grade, but it’s
enough to parse the torrent files generated by Gazelle lol.
This should help with … I haven’t really figured out yet what it helps
with I guess. But it was fun. I like the `Parse` abstraction very
much. It can parse XML and it can parse Bencode. Good.
Change-Id: If1331de423eab3e91ce9db6e2a7eb84da51b18a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13211
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Experiment: can we convert any Html-producing conduit into a
self-updating htmx snippet? Yes!
Add the resulting handler and initial snippet, and it will poll the
handler every second or so until the conduit stops emitting values.
Change-Id: I0bed17a5db8691a19df6c334ae1268dd6a3d3123
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13209
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The unpacked lrclib dump is 40GB at this point, but compresses very
well (to about 10GB give or take). So I thought about how to store it
in a compressed on disk, but still allow sqlite to query
efficiently (i.e. use a compression format that has arbitrary seek
indices).
Squashfs! with zstd compression!
```
mksquashfs ~/.cache/lyric-bak/ \
~/tmp/squashtest \
-comp zstd \
-Xcompression-level 5
```
Still need to write a helper that does the download & compression for
me to update the database once in a blue moon.
Unfortunately, sqlite-utils does not allow URL connection strings yet,
so I had to do a horrible SQL interpolation bit.
Issue tracker: https://mastodon.xyz/@Profpatsch/114083481108799334
This also changes the query to filter out every track that has neither
plain nor synched lyrics (yeah those exist for some reason?).
Change-Id: Ief0519ae8d9e5f14b66f6df6e3bcecae911ea142
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13208
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
For now just a setting whether we want to use freeleech tokens.
Change-Id: I1c0228031df8c79c2ec26ec5bdfef6dde1cb373e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13007
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
A generic way to turn Haskell structs into json objects, and therefor
to otel attributes.
I don’t know how to make it so I can give anything where Rep
implements that thing a `toOtelJsonAttr` implementation, things
overlap too much if I do that …
Change-Id: Iededb697dff206ddc6481a2eead210d44faa70c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13006
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
I got stuck with unsolvable type inference problems using HasField and
a struct here, since this is all returning an enum anyway, why not
just use a GADT. It seems to work just fine and is more
ergonomic (still not very, but this is Haskell).
Change-Id: I1b5d0f98528ff85c0e3044bc730b660972142cd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13005
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This is kind of a chonker because I went into so many rabbit holes.
Foremost this implements a simple “Refresh Artist” button that fetches
current artist torrent groups.
BUG: the `artist` endpoint torrent struct is shite, it’s missing most
info that we get in the `search` endpoint torrent struct, plus it’s
organized differently (e.g. the `artists` thingy is in the
torrent_group not the torrent).
I should switch everything over to fetching the `torrent_group.id`s
first and then going through and slowly fetching every torrent group
separately … however that might time out very quickly. ugh. There
doesn’t seem to be a way of fetching multiple torrent groups.
Random other shit & improvements:
* intersperse for builders
* fix json errors so that the structs don’t get too
big (`restrictJson`)
* show error messages as json so jaeger displays it with nested UI
* color pretty-printed json outpt on command line
* add some important integral functions to MyPrelude
* add `sintersperse` and `mintersperse` to MyPrelude
Change-Id: If8bfcd68dc5c905e118ad86d50d7416962bf55d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12960
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
We can’t do more than like 5 requests right next to each other, so
let’s handle the timeout they request.
This kinda destroys search speeds for large search results,
so we might have to filter out collections somehow, or do something
smarter like schedule things out and show a preliminary result at one
point.
Change-Id: If916379eb6e19cf8e960cf7553965b338645e560
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12958
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This makes `addAttribute` a little nicer to handle, because we can now
just put a `(builder, value)` tuple (yay, orphan instances!)
Change-Id: I145a0d2b764d44755fb3c548a40188f13ee4ed83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12956
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
I’ve had enough of `show & stringToText` bullshit, let’s finally
create a trivial builder contravariant module that wraps
`Text.Builder` and `ByteString.Builder` and has a naming scheme that
does not collide with anything and that I can remember in my sleep.
Plus some experiments in making `Divisible` useful with `HasField`.
Change-Id: Ie25f78226d24eadf4c35353fe9db40b7683d82c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12955
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Apparently they added the ability to add random files (e.g. pdfs), and
the API returns undocumented objects if that happens.
Let’s skip these.
Change-Id: Icd783a6ed2114520e5c524f2a2c3acfcb67d792e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12954
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Small helper class for putting a json otel attribute from random
types, via Enc.
Used for the redacted requests for now.
Change-Id: I29c31de01f1f5eb3f63ce5639e5b4df4f9b0dc40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12953
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The json parsing library gives us an error path where the parse
failed, which means we can index into the path to show the json value
that failed us.
This can be quite expensive (and large!) of course, but the error
message clarity is worth it methinks.
Change-Id: Icacbd799254aaecd4a939ca13e6070d68a78138d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12952
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
When searching for an artist, we only put albums before, but now let’s
fetch all releases and group them by release type.
I group & sort on the backend instead of the database, cause it’s a
lot easier to express in Haskell code and the amount of data stays the
same (except for the filter for singles which I might move into the
SQL at one point?)
Adds a prelude module for better dealing with comparisons.
Change-Id: Iff235af611b6e1bac71b118a8a04fc73cacd169f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12951
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
For some reason the driver does not mount the bus sometimes, in that
case just call the tool directly without a specific bus, so we get at
least some sort of brightness adjustment (lol).
Change-Id: Ie8fe8c500fb1025609b569715e681e053e6e06ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12909
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
For trying out a bunch of themes in quick succession.
Also note down a few of the cooler themes.
Change-Id: I0da80cc0945dba03d3592f28f4c34df4b5969e82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12893
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This simple dbus service will use the ddcci interfaces to change
brightness for both the internal and external monitor (roughly in
sync).
Currently in the alacritty-change-color-scheme script because I’m lazy
and still experimenting.
Change-Id: Ib2c4323699ed9d19ee398f84680b755df4b25798
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12891
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
These `tsconfig` will apply to JS files with jsdoc comments. Sweet!
Change-Id: I1a623d0ec7e2d73e99d7c6aaf8162d96f4ff2b29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12889
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
We want to use the tracers quite similar to how we’d use OTEL for
tracing, meaning you should be able to start a span and use it within
another span to register it as a parent span.
They are also batched up and sent asynchrously, so the won’t incur a
lot of overhead on dbus nor block the main execution flow (done via
sending a nodejs event and a dedicated batch sending process).
Change-Id: If61b85305807e661ffee386f793c11c4b7a858a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12888
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
This makes defining the interface a little less verbose & more
typesafe (checks are done by the dbus library).
Change-Id: I16df987fd152cabf76ed9878ed1a372a0f7003fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12886
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
For simplicity’s sake this puts everything into the
alacritty-change-color-scheme script for now.
This implements a simple dbus-opentelemetry proxy adapter, which
allows services to record otel traces without having to depend on the
quite complex otel libraries. Instead, they just send their traces to
the dbus tracing interface, and the service that binds against that
interface forwards the spans to the OTLP collector.
First you create a new Tracer for your service via the `TracerFactory`
interface:
```
> busctl --user call \
de.profpatsch.otel.Tracer \
/de/profpatsch/otel/TracerFactory \
de.profpatsch.otel.TracerFactory CreateTracer \
s hello
s "/de/profpatsch/otel/tracers/hello"
```
(this corresponds to setting up a tracer with properties in OTEL)
Then, you can use the returned object path to call the `Tracer`
interface proper:
```
< busctl --user call \
de.profpatsch.otel.Tracer \
/de/profpatsch/otel/tracers/hello \
de.profpatsch.otel.Tracer \
StartSpan \
s '{"spanId": "111", "name": "111"}'
```
This will create the spans. You can also set their timestamps on the
sending side via `startTime`/`endTime`, but make sure it’s a hrtime
tuple.
Prefer batching multiple spans vie the `BatchSpans` call.
Change-Id: Ie6cfdcb0dc3e2398316a2c1763bc72c1118168b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12885
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Adds a simple-stupid dbus interface for this daemon which allows
on-the-fly changing of the alacritty color scheme.
Example call:
```
busctl --user call de.profpatsch.alacritty.ColorScheme \
/de/profpatsch/alacritty/ColorScheme \
de.profpatsch.alacritty.ColorScheme \
SetColorScheme s 'prefer-dark'
```
Change-Id: Ic895fedefb3f5bd95f2279edf53fe179e8f24f89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12875
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
they switched from .ch to .sh
Change-Id: I889634ec257b7956b9d2b22a9ad6fc0c889f43c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12853
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Uses xapian under the hood to index the contents, then makes it
searchable with a CGI binary on http://localhost:8080
We could in theory index every -doc output this way to get local
documentation search for the current system (similar to `man-db`).
Change-Id: I2588c8f100841cfbed570bb65d376b79747c06ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12710
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
