When running using async, there are a number of py...
# prefect-community
When running using async, there are a number of pythonic ways to approach concurrency limits including queues and semaphores. The examples I have seen with Prefect 2 (and I may not be looking hard enough) are associated with process concurrency and not async concurrency. Is there a recommended way of limiting async concurrency? Are we free to use
and the like? Thanks for any suggestions or pointers.
Hi Sean - I’m not really sure I quite understand exactly what you are trying to accomplish. The behavior for the concurrency limits should be at the deployment level on the work-queue - what are you looking to accomplish as this seems like it’s trying to re-invent the concurrency already implemented into prefect?
I would recommend using anyio's task group and concurrency primitives.
Although I'm not actually sure if those will work as intended when using the concurrent task runner. There was a recent pull request to add worker limits to the concurrent task runner but they ran into problems and ended up using our tag based concurrency limits instead.
@Christopher Boyd I think the problem I am trying to solve is a common one. I'd like to run, for example, 10 concurrent async tasks using one process (concurrency 1, in Prefect terms). @Zanie I'll take a look at the
stuff. I've generally been using straight asyncio, but using some of the
higher-level concepts might clarify things. For background, I'm scraping APIs and a surprising amount of creative hand-tuning of async and concurrency (process-based for transformation) is required for each to be performant and idempotent.
Prefect’s concurrency limiting isn’t process/thread based it’s just a global limit on concurrent tasks with a given label.