Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

When running using async, there are a number of pythonic ways to approach concurrency limits including queues and semaphores. The examples I have seen with Prefect 2 (and I may not be looking hard enough) are associated with process concurrency and not async concurrency. Is there a recommended way of limiting async concurrency? Are we free to use `asyncio.Semaphore` or `asyncio.Queue` and the like? Thanks for any suggestions or pointers.

Hi Sean - I’m not really sure I quite understand exactly what you are trying to accomplish.
The behavior for the concurrency limits should be at the deployment level on the work-queue - what are you looking to accomplish as this seems like it’s trying to re-invent the concurrency already implemented into prefect?

I would recommend using anyio's task group and concurrency primitives.

Although I'm not actually sure if those will work as intended when using the concurrent task runner. There was a recent pull request to add worker limits to the concurrent task runner but they ran into problems and ended up using our tag based concurrency limits instead.

<@U03FKQX1TH9> I think the problem I am trying to solve is a common one. I'd like to run, for example, 10 concurrent async tasks using one process (concurrency 1, in Prefect terms). <@U01CEUST9B5> I'll take a look at the `anyio` stuff. I've generally been using straight asyncio, but using some of the `anyio` higher-level concepts might clarify things. For background, I'm scraping APIs and a surprising amount of creative hand-tuning of async and concurrency (process-based for transformation) is required for each to be performant and idempotent.

Prefect’s concurrency limiting isn’t process/thread based it’s just a global limit on concurrent tasks with a given label.