Andrew Lawlor
05/16/2022, 5:15 PMCreating a new Dask cluster with `__main__.get_executor.<locals>.<lambda>`...`
in my gke logs, im seeing
raise TimeoutError( asyncio.exceptions.TimeoutError: Nanny failed to start in 60 seconds
Kevin Kho
05/16/2022, 5:40 PMAndrew Lawlor
05/16/2022, 5:50 PMDaskExecutor(
cluster_class=lambda: KubeCluster(make_pod_spec(image=prefect.context.image), env=env),
adapt_kwargs={"minimum": 2, "maximum": 3},
)
so a new temporary cluster gets spun up with each flow, which seems to be the preferred way to run a flow with a dask executor. is there a recommended way to set up the temporary cluster?Kevin Kho
05/16/2022, 5:55 PMAndrew Lawlor
05/16/2022, 5:59 PMKevin Kho
05/16/2022, 6:02 PM