https://prefect.io logo
Powered by
Title
b

Ben Welsh

01/31/2022, 1:56 PM
My next challenge trying to configure my Prefect 
LocalDaskExecutor
to run mapped tasks in parallel inside a single GKE k8s container with what, if I understand the jargon correctly, is known as vertical scaling. Are there any resources that show the optimal way to do that?
a

Anna Geller

01/31/2022, 1:59 PM
This post talks more about it https://rdrn.me/scaling-out-prefect/
b

Ben Welsh

01/31/2022, 2:25 PM
Thanks. The issue I'm currently encountering is that my Prefect tasks are running one by one in the cloud, despite being mapped. I suspect, but I'm far from sure, that this is due to a lack of CPU in my cloud deployment. My autopilot GKE cluster is spinning up a single job container for each Prefect task, with 0.5 CPU assigned. Is getting that CPU request elevated at GKE the core problem I need to solve, or something else?
For instance, I see these settings in my agent config
- name: JOB_CPU_REQUEST
value: ''
- name: JOB_CPU_LIMIT
value: ''
I inserted "2000m" into those values, thinking that would spin up job containers with 2 cores, and then ran kubectl apply to reconfigre my agent. That didn't seem to do it. The next job I created still had a request for 500m CPU. I feel like I'm close but not quite getting it.
a

Anna Geller

01/31/2022, 2:26 PM
can you share your executor and run config?
You can influence how many cores are used by setting the num_workers: 
# Use 8 threads
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)

# Use 8 processes
flow.executor = LocalDaskExecutor(scheduler="processes", num_workers=8)
I think you’re definitely on the right path: you can combine the settings on the executor and on the run config to get the desired resource utilization: 
with Flow(
        FLOW_NAME,
        storage=STORAGE,
        run_config=KubernetesRun(
            labels=["k8s"],
            cpu_request=0.5,
            memory_request="2Gi",
        ),
) as flow:
b

Ben Welsh

01/31/2022, 3:35 PM
Ah. So the request number should go in the flow, perhaps, and not in the agent config.
a

Anna Geller

01/31/2022, 3:41 PM
correct, this way you can override whatever is set on the agent
b

Ben Welsh

01/31/2022, 4:43 PM
I think that did it. Thanks for your help.
👍 1
#prefect-serverJoin Slack
Powered by