Limiting Concurrency in Aff

Help
#1

Is there an idiomatic way to limit the number of aff fibers running for a particular task?

For context, I’m writing some code to backup a podcast from an rss feed (just for my own education), and to download them I’m doing something like this.

downloadAll = do
   rss <- getFeed
   rss.items # parTraverse_ downloadItem

This starts an http request immediately for each item, which seems unfriendly to the server.

Is there an idiomatic way to queue the Affs so that only n of them are running at any one time?

Right now I’m futzing around with storing a queue in an AVar, but I have no idea what I’m doing and it feels like something there might just be a library method for?

#2

You could use a BoundedQueue from https://github.com/purescript-contrib/purescript-concurrent-queues, store a number of Unit's in it and then use it a like a semaphore. Every request first needs to pop a value of the queue, and when it’s done it pushes a Unit back onto the queue.

1 Like
#3

Actually it might be even better to just use the capacity… so every request pushes a Unit into an initially empty BoundedQueue, and pops a value after it’s done. Then you can make sure the Queue is empty after you’re done so you don’t have any outstanding requests.

1 Like
#4

Thanks for the help, I’ll give that a shot

#5

I haven’t run this :laughing: but it might do the job in a pinch using AVar as a queue.

parTraverseBounded :: forall f a b. Traversable f => Int -> (a -> Aff b) -> f a -> Aff (f b)
parTraverseBounded max k xs = do
  avail <- AVar.empty
  let done = const $ liftEffect $ EffectAVar.put unit avail mempty
  for_ (1 .. max) done
  parTraverse (bracket (AVar.take avail) done <<< const <<< k) xs
1 Like
#6

Running code before hitting send is for cowards