Is there such thing as task concurrency limits? I have 8 or so tasks all running with

.submit

with the

DaskTaskRunner

with no cluster. It seems like about two tasks run at a time and then another two start, my assumption was that all 8 would start and Dask handles them and they finish when they are ready? I know this would be limited by the threads too, but the machine it is running on should have more than enough threads to handle those 8 tasks.

I am not sure if there is a way of specifying the limit through some task's

.submit

interface. However, as a work around, you could consider providing the specifications to a dask schedular yourself to the

DaskTaskRunner

so that there is an explicit number of dask workers spawned. I do this with a

dask_jobqueue.SLURMCluster

Thanks @Tim Galvin, any code examples of this to get me started?

Off hand no sorry. I am using a SLURMCluster, and it depends on the type of the Dask distributed schedular you are intending to run. Is it just one on your local system that you are interested in?

You can also specify concurrency limits based on tags applied to a task https://docs.prefect.io/concepts/tasks/#task-run-concurrency-limits