Ben Muller
12/05/2022, 6:15 AM.submit
with the DaskTaskRunner
with no cluster.
It seems like about two tasks run at a time and then another two start, my assumption was that all 8 would start and Dask handles them and they finish when they are ready?
I know this would be limited by the threads too, but the machine it is running on should have more than enough threads to handle those 8 tasks.Tim Galvin
12/05/2022, 6:19 AM.submit
interface. However, as a work around, you could consider providing the specifications to a dask schedular yourself to the DaskTaskRunner
so that there is an explicit number of dask workers spawned. I do this with a dask_jobqueue.SLURMCluster
Ben Muller
12/05/2022, 6:35 AMTim Galvin
12/05/2022, 8:52 AMMason Menges
12/06/2022, 5:31 PM