BK Lau
01/20/2021, 8:55 PMJim Crist-Harif
01/20/2021, 9:00 PMDaskExecutor
. This would let dask manage dispersing tasks across your cluster, with all flow-runners (lightweight processes managing a flow run) running on a single node with the agent. This works well for task level parallelism, but less well for flow level paralellism
Which option is best depends on your use case. You could also mix these, with a few nodes running agents, and the remaining running a dask cluster.BK Lau
01/20/2021, 9:46 PMJim Crist-Harif
01/20/2021, 9:49 PMBK Lau
01/25/2021, 11:04 PMJim Crist-Harif
01/26/2021, 1:46 AMBK Lau
01/26/2021, 2:44 PMMarz
04/25/2021, 8:19 PMBK Lau
04/28/2021, 3:51 PM