When running using async, there are a number of py...
# prefect-community
s
When running using async, there are a number of pythonic ways to approach concurrency limits including queues and semaphores. The examples I have seen with Prefect 2 (and I may not be looking hard enough) are associated with process concurrency and not async concurrency. Is there a recommended way of limiting async concurrency? Are we free to use
asyncio.Semaphore
or
asyncio.Queue
and the like? Thanks for any suggestions or pointers.
1
c
Hi Sean - I’m not really sure I quite understand exactly what you are trying to accomplish. The behavior for the concurrency limits should be at the deployment level on the work-queue - what are you looking to accomplish as this seems like it’s trying to re-invent the concurrency already implemented into prefect?
z
I would recommend using anyio's task group and concurrency primitives.
Although I'm not actually sure if those will work as intended when using the concurrent task runner. There was a recent pull request to add worker limits to the concurrent task runner but they ran into problems and ended up using our tag based concurrency limits instead.
s
@Christopher Boyd I think the problem I am trying to solve is a common one. I'd like to run, for example, 10 concurrent async tasks using one process (concurrency 1, in Prefect terms). @Zanie I'll take a look at the
anyio
stuff. I've generally been using straight asyncio, but using some of the
anyio
higher-level concepts might clarify things. For background, I'm scraping APIs and a surprising amount of creative hand-tuning of async and concurrency (process-based for transformation) is required for each to be performant and idempotent.
z
Prefect’s concurrency limiting isn’t process/thread based it’s just a global limit on concurrent tasks with a given label.