When running using async, there are a number of pythonic ways to approach concurrency limits including queues and semaphores. The examples I have seen with Prefect 2 (and I may not be looking hard enough) are associated with process concurrency and not async concurrency. Is there a recommended way of limiting async concurrency? Are we free to use

asyncio.Semaphore

or

asyncio.Queue

and the like? Thanks for any suggestions or pointers.

Hi Sean - I’m not really sure I quite understand exactly what you are trying to accomplish. The behavior for the concurrency limits should be at the deployment level on the work-queue - what are you looking to accomplish as this seems like it’s trying to re-invent the concurrency already implemented into prefect?

I would recommend using anyio's task group and concurrency primitives.

@Christopher Boyd I think the problem I am trying to solve is a common one. I'd like to run, for example, 10 concurrent async tasks using one process (concurrency 1, in Prefect terms). @Zanie I'll take a look at the

anyio

stuff. I've generally been using straight asyncio, but using some of the

anyio

higher-level concepts might clarify things. For background, I'm scraping APIs and a surprising amount of creative hand-tuning of async and concurrency (process-based for transformation) is required for each to be performant and idempotent.

Prefect’s concurrency limiting isn’t process/thread based it’s just a global limit on concurrent tasks with a given label.