feat(users/Profpatsch/whatcd-resolver): conduit shenanigans

Some experiments in speeding up search efficiency.

This is more of a “is this possible” thing, and it looks like it’s
possible, but does not really improve anything regarding the rate
limiting.

The idea is that we can start everything at the same time as async,
but use a semaphore to have only 5 things run at once. That also means
that as soon as something is done, we immediately start the next task.
The asyncs are guaranteed to be cleaned up by the `ResourceT`
wrapper (eventually).

It’s pretty cool how Conduit makes writing these side-effecting things
pretty possible.

Change-Id: Ibadead7db293373b415840960602fa71920fc653
Reviewed-on: https://cl.tvl.fyi/c/depot/+/13246
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This commit is contained in:
Profpatsch 2025-03-11 18:42:31 +01:00
parent ae0e75aaf2
commit 42da189180
2 changed files with 156 additions and 21 deletions

View file

@ -89,6 +89,7 @@ library
pa-run-command,
aeson-better-errors,
aeson,
async,
bencode,
blaze-html,
bytestring,
@ -109,6 +110,7 @@ library
mtl,
network-uri,
random,
resourcet,
resource-pool,
template-haskell,
postgresql-simple,