Is there an idiomatic way to limit the number of aff fibers running for a particular task?
For context, I’m writing some code to backup a podcast from an rss feed (just for my own education), and to download them I’m doing something like this.
downloadAll = do
rss <- getFeed
rss.items # parTraverse_ downloadItem
This starts an http request immediately for each item, which seems unfriendly to the server.
Is there an idiomatic way to queue the
Affs so that only
n of them are running at any one time?
Right now I’m futzing around with storing a queue in an AVar, but I have no idea what I’m doing and it feels like something there might just be a library method for?
You could use a
BoundedQueue from https://github.com/purescript-contrib/purescript-concurrent-queues, store a number of
Unit's in it and then use it a like a semaphore. Every request first needs to pop a value of the queue, and when it’s done it pushes a
Unit back onto the queue.
Actually it might be even better to just use the capacity… so every request pushes a
Unit into an initially empty BoundedQueue, and pops a value after it’s done. Then you can make sure the Queue is empty after you’re done so you don’t have any outstanding requests.
Thanks for the help, I’ll give that a shot
I haven’t run this but it might do the job in a pinch using AVar as a queue.
parTraverseBounded :: forall f a b. Traversable f => Int -> (a -> Aff b) -> f a -> Aff (f b)
parTraverseBounded max k xs = do
avail <- AVar.empty
let done = const $ liftEffect $ EffectAVar.put unit avail mempty
for_ (1 .. max) done
parTraverse (bracket (AVar.take avail) done <<< const <<< k) xs
Running code before hitting send is for cowards