https://prefect.io
Join Slack
can anyone limit the number of threads/workers when using concurrent/dask task runner? marvin is giv...
# ask-community
t
can anyone limit the number of threads/workers when using concurrent/dask task runner? marvin is giving me an answer but I don't think it's quiote working https://prefect-community.slack.com/archives/C04DZJC94DC/p1718130193281339
n
for the concurrent task runner, you should just be able to use our concurrency context manager or a 
asyncio.Semaphore
yourself, for dask I'm not so sure - you might have to read more here about that
t
@Nate thank you for the tip on the context manager! that's on a task level right? there's no way to control threads on a flow level similar to the n workers cluster kwarg with dask?
n
not threads directly, but you can use the context manager to choke to some number of tasks, and since in 2.x tasks run in worker threads, thats effectively the same thing
t
ok, I think I'm following you. let me rewrite some stuff and we'll see how it goes. thanks for the tips as always nate!
is there a better way to control the number of threads on per deployment basis than setting an env var? would there be any issue with using an env var like that?
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by