Ben Welsh
01/31/2022, 1:56 PMLocalDaskExecutor
to run mapped tasks in parallel inside a single GKE k8s container with what, if I understand the jargon correctly, is known as vertical scaling. Are there any resources that show the optimal way to do that?Anna Geller
01/31/2022, 1:59 PMBen Welsh
01/31/2022, 2:25 PM- name: JOB_CPU_REQUEST
value: ''
- name: JOB_CPU_LIMIT
value: ''
Anna Geller
01/31/2022, 2:26 PM# Use 8 threads
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)
# Use 8 processes
flow.executor = LocalDaskExecutor(scheduler="processes", num_workers=8)
with Flow(
FLOW_NAME,
storage=STORAGE,
run_config=KubernetesRun(
labels=["k8s"],
cpu_request=0.5,
memory_request="2Gi",
),
) as flow:
Ben Welsh
01/31/2022, 3:35 PMAnna Geller
01/31/2022, 3:41 PMBen Welsh
01/31/2022, 4:43 PM