Preston Marshall
02/07/2020, 11:14 PMChris White
02/07/2020, 11:15 PMPreston Marshall
02/07/2020, 11:18 PMChris White
02/07/2020, 11:20 PMstart_cluster
(which appears to be a custom function) is creating a single worker with only 1 thread and 1 process availablePreston Marshall
02/07/2020, 11:21 PMfrom dask_kubernetes import KubeCluster
def start_cluster():
cluster = KubeCluster.from_yaml('worker-spec.yml')
cluster.scale_up(10) # specify number of nodes explicitly
cluster.adapt(minimum=1, maximum=100)
return cluster
Chris White
02/07/2020, 11:22 PMexecutor = DaskExecutor()
Preston Marshall
02/07/2020, 11:22 PMChris White
02/07/2020, 11:22 PMPreston Marshall
02/07/2020, 11:23 PMChris White
02/07/2020, 11:24 PMnthreads
and nprocs
?Preston Marshall
02/07/2020, 11:26 PMChris White
02/07/2020, 11:27 PMPreston Marshall
02/07/2020, 11:27 PMChris White
02/07/2020, 11:28 PMPreston Marshall
02/07/2020, 11:28 PMChris White
02/07/2020, 11:29 PMPreston Marshall
02/07/2020, 11:29 PMChris White
02/07/2020, 11:29 PMPreston Marshall
02/07/2020, 11:30 PMChris White
02/07/2020, 11:30 PMPreston Marshall
02/08/2020, 12:55 AMChris White
02/08/2020, 1:02 AM