https://prefect.io
Join Slack
My next challenge trying to configure my Prefect `LocalDaskExecutor` to run mapped tasks in parallel...
# prefect-server
b
My next challenge trying to configure my Prefect 
LocalDaskExecutor
to run mapped tasks in parallel inside a single GKE k8s container with what, if I understand the jargon correctly, is known as vertical scaling. Are there any resources that show the optimal way to do that?
a
This post talks more about it https://rdrn.me/scaling-out-prefect/
b
Thanks. The issue I'm currently encountering is that my Prefect tasks are running one by one in the cloud, despite being mapped. I suspect, but I'm far from sure, that this is due to a lack of CPU in my cloud deployment. My autopilot GKE cluster is spinning up a single job container for each Prefect task, with 0.5 CPU assigned. Is getting that CPU request elevated at GKE the core problem I need to solve, or something else?
For instance, I see these settings in my agent config
- name: JOB_CPU_REQUEST
value: ''
- name: JOB_CPU_LIMIT
value: ''
I inserted "2000m" into those values, thinking that would spin up job containers with 2 cores, and then ran kubectl apply to reconfigre my agent. That didn't seem to do it. The next job I created still had a request for 500m CPU. I feel like I'm close but not quite getting it.
a
can you share your executor and run config?
You can influence how many cores are used by setting the num_workers:
Copy code
# Use 8 threads
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)

# Use 8 processes
flow.executor = LocalDaskExecutor(scheduler="processes", num_workers=8)
I think you’re definitely on the right path: you can combine the settings on the executor and on the run config to get the desired resource utilization:
Copy code
with Flow(
        FLOW_NAME,
        storage=STORAGE,
        run_config=KubernetesRun(
            labels=["k8s"],
            cpu_request=0.5,
            memory_request="2Gi",
        ),
) as flow:
b
Ah. So the request number should go in the flow, perhaps, and not in the agent config.
a
correct, this way you can override whatever is set on the agent
b
I think that did it. Thanks for your help.
👍 1
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by