Some experiments in speeding up search efficiency.
This is more of a “is this possible” thing, and it looks like it’s
possible, but does not really improve anything regarding the rate
limiting.
The idea is that we can start everything at the same time as async,
but use a semaphore to have only 5 things run at once. That also means
that as soon as something is done, we immediately start the next task.
The asyncs are guaranteed to be cleaned up by the `ResourceT`
wrapper (eventually).
It’s pretty cool how Conduit makes writing these side-effecting things
pretty possible.
Change-Id: Ibadead7db293373b415840960602fa71920fc653
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13246
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This bunches up 5 search page requests to run at the same time.
We use a conduit now, so we could get smart about returning partial
results and such (if for example upstream puts us into the rate limit,
which they do after 10 requests.
Change-Id: Idbb174334fa499c16b3426a8d129deaf3a1d3b0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13245
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
I missed this! It speeds up `/artist` by another 50%.
Change-Id: I138c38185ea7fb606a139b436b9c956f7b8e1a37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13244
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
move the label stuff into its own temporary module (until we figure
out what to put into pa-label).
Also rewrite a few things to use t2/t3.
Change-Id: I8cc8678ec01a56d6c738eb4833a3ba566a7a1e20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13242
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
We need `artist_ids` in most of the queries, so let’s make them a
generated column and put an index on them.
This reduces times for getArtistNameById from ~300ms to ~20ms and for
the main “latest” table from ~250ms to ~60ms.
The `artist_has_been_snatched` logic moves into the torrent filter, so
we don’t have to work on such large records further down.
Change-Id: I5a1929bd9c81ea0031e512d01897c0e460ded077
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13240
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Or rather make it less slow lol.
I should put the artists in a computed column now.
Change-Id: I28f6546353e2930156eaeb427ba5122559aea49b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13239
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
A simple favourite system that adds an artist to your favourites when
you go to their artist page for the first time.
Also, if you fetch something from an artist, they get automatically
added to the favourites.
This is pretty much just done by adding more crap to the main SQL
query, and has the disadvantage that the `artist_has_been_snatched`
CTE slows down the query a lot for the favourite filter (by 50% or
even sometimes 150% in a weird degenerate case). I think we can
optimize this somewhat, but that needs a small rewrite of the query
CTE stages.
Change-Id: I7aa6d99dc26e24eb22ef56ffd4d2db5c6978ad48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13238
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
For the next step, we need to put more things in WITH-Clauses, so
let’s prepare it now to get a nicer diff.
Change-Id: Ibd5b67c77c87ce3d6aa0ab04ca3d66f9b90dc856
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13237
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
For latest releases, I really don’t want to see bootlegs and stuff.
Change-Id: I3a98d7b81d2ca7962dcef619d4dcfcbd18e98f84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13226
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Let’s start improving the main page.
So far, it was just all release groups sorted by weight on a single
page, which was not super helpful (and got kinda large).
The first feature is to show the latest releases that are known.
This is done by torrent group ID. This ID does not always correspond
to the date, but can also be a very old album that gets uploaded,
or (seldomly) a group that gets merged for metadata.
We should think about restricting this to favourites, automatically
marking everything as favourite where we have an album downloaded or
clicked on the artist before, and then selectively allow to un-mark
it after the fact.
An even stronger “not interested” could be used to automatically
reclaim seedbox space once it becomes an issue.
Eventually (after implementing favourites), we should introduce a
job system that automatically updates these entries every few hours.
Maybe even have a “very interested” feature that automatically
downloads everything new for an artist?
And then a “veryvery interested” feature that also buys the thing from
bandcamp lol
Change-Id: I467c350722279ff37150f847f5014d7e0e67e626
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13225
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The RED Gazelle API is so inconsistent, smh.
Sometimes they return the release type string, sometimes the integer
representing it. So let’s handle both!
Change-Id: Ie99f01342a13269df5c1e2036aaede54d7149516
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13223
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
For all big page reloads, we want the `<head>` of the page to start
being transmitted even while doing database requests.
So let’s use the `Wai.ResponseStream` to do exactly that. The handler
provides the contents of `<head>`, we start streaming that, meanwhile
it calculates the `<body>` and once that is ready transmits it.
This means we can load all our static resources before the page
even starts sending any body data, meaning the css and html is already
there when we reach `</html>`.
Sweet.
The `<title>` in `artistPage` was depending on the table data, so I
moved it into a separate SELECT.
We can do all of it in parallel as well. Sweet.
~~~
This also adds static file prefetching of source maps if provided.
Change-Id: Ib824430594733b4c8e86ee1096c8afba2df1a66d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13221
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
The `bencode` library is anything but production-grade, but it’s
enough to parse the torrent files generated by Gazelle lol.
This should help with … I haven’t really figured out yet what it helps
with I guess. But it was fun. I like the `Parse` abstraction very
much. It can parse XML and it can parse Bencode. Good.
Change-Id: If1331de423eab3e91ce9db6e2a7eb84da51b18a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13211
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
For now just a setting whether we want to use freeleech tokens.
Change-Id: I1c0228031df8c79c2ec26ec5bdfef6dde1cb373e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13007
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This is kind of a chonker because I went into so many rabbit holes.
Foremost this implements a simple “Refresh Artist” button that fetches
current artist torrent groups.
BUG: the `artist` endpoint torrent struct is shite, it’s missing most
info that we get in the `search` endpoint torrent struct, plus it’s
organized differently (e.g. the `artists` thingy is in the
torrent_group not the torrent).
I should switch everything over to fetching the `torrent_group.id`s
first and then going through and slowly fetching every torrent group
separately … however that might time out very quickly. ugh. There
doesn’t seem to be a way of fetching multiple torrent groups.
Random other shit & improvements:
* intersperse for builders
* fix json errors so that the structs don’t get too
big (`restrictJson`)
* show error messages as json so jaeger displays it with nested UI
* color pretty-printed json outpt on command line
* add some important integral functions to MyPrelude
* add `sintersperse` and `mintersperse` to MyPrelude
Change-Id: If8bfcd68dc5c905e118ad86d50d7416962bf55d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12960
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This makes `addAttribute` a little nicer to handle, because we can now
just put a `(builder, value)` tuple (yay, orphan instances!)
Change-Id: I145a0d2b764d44755fb3c548a40188f13ee4ed83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12956
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Apparently they added the ability to add random files (e.g. pdfs), and
the API returns undocumented objects if that happens.
Let’s skip these.
Change-Id: Icd783a6ed2114520e5c524f2a2c3acfcb67d792e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12954
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Small helper class for putting a json otel attribute from random
types, via Enc.
Used for the redacted requests for now.
Change-Id: I29c31de01f1f5eb3f63ce5639e5b4df4f9b0dc40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12953
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
When searching for an artist, we only put albums before, but now let’s
fetch all releases and group them by release type.
I group & sort on the backend instead of the database, cause it’s a
lot easier to express in Haskell code and the amount of data stays the
same (except for the filter for singles which I might move into the
SQL at one point?)
Adds a prelude module for better dealing with comparisons.
Change-Id: Iff235af611b6e1bac71b118a8a04fc73cacd169f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12951
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
they switched from .ch to .sh
Change-Id: I889634ec257b7956b9d2b22a9ad6fc0c889f43c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12853
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
I changed the seeding weight, so now it would not show already
downloaded torrents, prompting me to fetch stuff that I already have
in new qualities. Obviously, that’s not the best idea.
Kinda surprised it’s this easy to fix heh.
Change-Id: I6bbf325672a91d794d144c006ccf3d702a581bce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12330
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
AppException would be a console-pretty-printed version for http
errors, which would print all the escape codes in the jaeger traces of
the exception, making it more-or-less unreadable.
So instead, let’s make AppException two cases, an ErrorTree case which
is printed as-is (no color), and a “Pretty” case which is printed
using the pretty module (colors on console, no colors in otel).
Somewhat involved, I guess this is temporary until I figure out what
is really needed.
Change-Id: Iff4a8651c5f5368a5b798541efc19cc7ab9de34b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12232
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
We’d transfer the full json data for each torrent from the db instead
of just the 2 or 3 fields we need.
Adds some more helpers for parsing database values.
Adds some better logging events & traces.
Change-Id: I5db386c4ea247febf5f9fc3815da2e7f11286d41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12140
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
It turns out the pg_format thing is just too slow for my use-cases
most of the time, even when pooling the mf. Most queries stay 90%+ in
the perl script, even though they are very fast to execute on their
own, screwing up the traces a lot.
So instead I replace the `postgres-simple` quasi-quoter that strips
whitespace (and tends to screw up queries anyway) with a simple one
that just removes the outer indentation up to the first line.
Why did I spend so much time on pg_format haha
Change-Id: I911cd869deec68aa5cf430ff4d111b0662ec6d28
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12138
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Ideally there’d be a better generic abstraction of doing basic http
calls (with tracing) in the future, but for now just reexport.
Change-Id: Id7548739ea62e9172f2773f8db79fe726096b7f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12136
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
There’s a bunch of duplication in how http client things are done,
let’s move that all to a single module.
Change-Id: Ic08c9bce49d562e4fa640a5bdfc15973a28a7bcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12135
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
The redacted URL needs the torrent group id to go to the album page,
not the torrent id.
Also open the external link in a separate tab.
Change-Id: Ie6d73cb02137855bb388859d2800968cd56ece49
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11745
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
We’re using the redacted id, not the database id.
This is more stable.
Change-Id: I4d24e5bc6cc469d811cb1402a83157359576ef77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11683
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
This took a while to figure out, but essentially sorting a thing with
one million queries takes a long time, as compared to doing it on a
small subset of things and then joining against the final ordering.
The generated column helps, too.
Change-Id: I1bf283e2be060748eebda92576e3d062c51a6777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11681
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Simple artist album page that only shows albums the artist was
involved with.
Change-Id: Icff34afc6d1b39b6fb17765c1b3ea500dd4b4d95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11675
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
apparently the `torrent_group` json only contains one artist, while
the torrent itself contains a list of them.
This is important, because we need the artist id and the
`torrent_group` does not provide it, only the torrent one.
Change-Id: I3db45f454f14e89ea8c8dafba2065ecd55f5bcda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11670
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
When showing the best torrents table, we want to be able to filter for
“only downloaded”.
Change-Id: Ibfe9212f3d422d305f858fbef023ee985c1183d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11650
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Move the http calls into their own module, so we can trace the request
and provide a simple copy-to-replay command.
We have to work around a bug in the otel library, which would limit
our attribute value length to 128 bytes because it uses the wrong
option value.
~~~
`ifExists` is finally made more useful for dealing with optional
attributes in e.g. lists.
Change-Id: Iafab523e9ec4b00136db43f31fdc12aeefb7f77c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11241
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Move the functionality into two coarse modules.
There’s still the question about whether functions that change the
database tables should be in their own storage module, but let’s see
if it gets too confusing.
Change-Id: Ied1d47b353dd4597ffea35f111f440aad22e981d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11238
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
