Ahmed Ezzat
12/21/2021, 10:06 AMflow.executor = prefect.executors.DaskExecutor(
cluster_class=lambda: KubeCluster(
pod_template=make_pod_spec(
memory_request="64M",
memory_limit="4G",
cpu_request="0.5",
cpu_limit="8",
threads_per_worker=24,
image=prefect.context.image,
),
deploy_mode="remote",
idle_timeout="0",
scheduler_service_wait_timeout="0",
env=dict(os.environ)
| {
"DASK_DISTRIBUTED__WORKER__MULTIPROCESSING_METHOD": "fork",
"DASK_DISTRIBUTED__SCHEDULER__ALLOWED_FAILURES": "100",
},
),
adapt_kwargs={"minimum": min_workers, "maximum": max_workers},
)
Anna Geller
12/21/2021, 10:29 AMLocalDaskExecutor
or does it only happen with Dask?Ahmed Ezzat
12/21/2021, 4:03 PMLocalDaskExecutor
2. Yes, my flattened tasks return a list I wouldn't say it's a big list as it's only around 1000-2000 item and all of them is numbers between 1-100Anna Geller
12/21/2021, 4:13 PMAhmed Ezzat
12/21/2021, 5:40 PMAnna Geller
12/21/2021, 5:53 PMAhmed Ezzat
12/21/2021, 6:03 PMAnna Geller
12/21/2021, 6:04 PMAhmed Ezzat
12/22/2021, 4:03 AM"DASK_DISTRIBUTED__SCHEDULER__WORK_STEALING": "false"
and it seems to produce more stable runs. the current run will take around 2-3 hours I'll keep you posted.Anna Geller
12/22/2021, 10:50 AMAhmed Ezzat
12/22/2021, 11:52 AMAnna Geller
12/22/2021, 12:07 PM