Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

can anyone limit the number of threads/workers when using concurrent/dask task runner? marvin is giving me an answer but I don't think it's quiote working <https://prefect-community.slack.com/archives/C04DZJC94DC/p1718130193281339>

for the concurrent task runner, you should just be able to use our concurrency context manager or a `asyncio.Semaphore` yourself, for dask I'm not so sure - you might have to read more <https://cloudprovider.dask.org/en/latest/|here> about that

<@U02GMEZU18B> thank you for the tip on the context manager! that's on a task level right? there's no way to control threads on a flow level similar to the n workers cluster kwarg with dask?

not threads directly, but you can use the context manager to choke to some number of tasks, and since in 2.x tasks run in worker threads, thats effectively the same thing

ok, I think I'm following you. let me rewrite some stuff and we'll see how it goes. thanks for the tips as always nate!

is there a better way to control the number of threads on per deployment basis than setting an env var? would there be any issue with using an env var like that?