My next challenge trying to configure my Prefect `LocalDaskE Prefect Community #prefect-server

My next challenge trying to configure my Prefect `...

Ben Welsh

01/31/2022, 1:56 PM

My next challenge trying to configure my Prefect

LocalDaskExecutor

to run mapped tasks in parallel inside a single GKE k8s container with what, if I understand the jargon correctly, is known as vertical scaling. Are there any resources that show the optimal way to do that?

Anna Geller

01/31/2022, 1:59 PM

This post talks more about it https://rdrn.me/scaling-out-prefect/

Ben Welsh

01/31/2022, 2:25 PM

Thanks. The issue I'm currently encountering is that my Prefect tasks are running one by one in the cloud, despite being mapped. I suspect, but I'm far from sure, that this is due to a lack of CPU in my cloud deployment. My autopilot GKE cluster is spinning up a single job container for each Prefect task, with 0.5 CPU assigned. Is getting that CPU request elevated at GKE the core problem I need to solve, or something else?

Ben Welsh

01/31/2022, 2:25 PM

For instance, I see these settings in my agent config

Ben Welsh

01/31/2022, 2:26 PM

- name: JOB_CPU_REQUEST

value: ''

- name: JOB_CPU_LIMIT

value: ''

Ben Welsh

01/31/2022, 2:26 PM

I inserted "2000m" into those values, thinking that would spin up job containers with 2 cores, and then ran kubectl apply to reconfigre my agent. That didn't seem to do it. The next job I created still had a request for 500m CPU. I feel like I'm close but not quite getting it.

Anna Geller

01/31/2022, 2:26 PM

can you share your executor and run config?

Anna Geller

01/31/2022, 2:27 PM

You can influence how many cores are used by setting the num_workers:

Copy code

# Use 8 threads
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)

# Use 8 processes
flow.executor = LocalDaskExecutor(scheduler="processes", num_workers=8)

Anna Geller

01/31/2022, 2:29 PM

I think you’re definitely on the right path: you can combine the settings on the executor and on the run config to get the desired resource utilization:

Copy code

with Flow(
        FLOW_NAME,
        storage=STORAGE,
        run_config=KubernetesRun(
            labels=["k8s"],
            cpu_request=0.5,
            memory_request="2Gi",
        ),
) as flow:

Ben Welsh

01/31/2022, 3:35 PM

Ah. So the request number should go in the flow, perhaps, and not in the agent config.

Anna Geller

01/31/2022, 3:41 PM

correct, this way you can override whatever is set on the agent

Ben Welsh

01/31/2022, 4:43 PM

I think that did it. Thanks for your help.

👍 1

Open in Slack

Previous Next