Question on Dask work stealing, mapped task distribution, and adaptive mode in the following scenario:
• Simple Flow with a single task that maps over items, e.g. 50 items
• Long-running Dask cluster in adaptive mode with a minimum of 1 worker always running and work stealing disabled
• Kick off a Flow run
• One mapped task (i.e. a mapped task for 1 item) begins running on the 1 worker that is immediately available
• Other workers begin to start up (takes several minutes since we use k8s cluster autoscaler to request new spot instances in AWS)
• Once new workers are available they don't seem to get allocated tasks and we notice that only one worker is busy
Do we need to ensure Dask workers are already running and available prior to starting a Flow with a single mapped task? Said another way, does disabling work stealing prevent mapped tasks (that haven't started running) from being allocated to Dask workers that started after the mapped tasks have been submitted to the scheduler?