https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> Is it possible to limit the number of...
# ask-marvin
s
@Marvin Is it possible to limit the number of concurrent tasks running in a 
ConcurrentTaskRunner
?
n
i need to revive marvin - sorry about that
yeah there's a couple ways to do concurrency
s
Oh man I hope he is okay 🙂
💙 1
n
s
So my main problem is I can’t use a global concurrency limit because I want it to only apply for the current flow that I am in.
I need max 20 of a specific task to run for a specific flow run. But its fine that 10 flows are running concurrently for a total of 200.
n
what about tag based concurrency
s
my understanding is that is the same problem because the tag based concurrency applies globally also
n
> So my main problem is I can’t use a global concurrency limit because I want it to only apply for the current flow that I am in. and well actually, you're free to create your own "namespaces" with the name so it would be essentially scoped to your specific flow
s
right so I suppose I could have the flow setup its own concurrency limit with a dynamically generated tag and use that
n
yeah - that's probably how i would go about it
s
Thank you!
n
👍
s
One last question - can I have a subflow use the task runner of the parent flow. For example I have a parent flow running in a local dask cluster and I want the subflow to run its tasks in the same cluster.
n
the subflow wouldn’t inherit the task runner by default, but you’d be free to specify the same task runner for each flow (subflow or not) that should use that task runner
s
wouldn’t that create a problem with a local dask runner because they are both trying to have a dask scheduler listening on the same port?
but maybe that is configurable actually
Just to tie a bow on this. Dask is smart enough to figure this out. It detects the port is already in use and picks another one. So the main concern would be resource constraints with running two dask clusters if one was to adopt this approach.
Thanks for you help Nate! I hope Marvin has a speedy recovery marvin
👍 1
n
💙
Open in Slack
PreviousNext
Powered by