b

    Ben Welsh

    7 months ago
    My next challenge trying to configure my Prefect
    LocalDaskExecutor
    to run mapped tasks in parallel inside a single GKE k8s container with what, if I understand the jargon correctly, is known as vertical scaling. Are there any resources that show the optimal way to do that?
    Anna Geller

    Anna Geller

    7 months ago
    This post talks more about it https://rdrn.me/scaling-out-prefect/
    b

    Ben Welsh

    7 months ago
    Thanks. The issue I'm currently encountering is that my Prefect tasks are running one by one in the cloud, despite being mapped. I suspect, but I'm far from sure, that this is due to a lack of CPU in my cloud deployment. My autopilot GKE cluster is spinning up a single job container for each Prefect task, with 0.5 CPU assigned. Is getting that CPU request elevated at GKE the core problem I need to solve, or something else?
    For instance, I see these settings in my agent config
    - name: JOB_CPU_REQUEST
    value: ''
    - name: JOB_CPU_LIMIT
    value: ''
    I inserted "2000m" into those values, thinking that would spin up job containers with 2 cores, and then ran kubectl apply to reconfigre my agent. That didn't seem to do it. The next job I created still had a request for 500m CPU. I feel like I'm close but not quite getting it.
    Anna Geller

    Anna Geller

    7 months ago
    can you share your executor and run config?
    You can influence how many cores are used by setting the num_workers:
    # Use 8 threads
    flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)
    
    # Use 8 processes
    flow.executor = LocalDaskExecutor(scheduler="processes", num_workers=8)
    I think you’re definitely on the right path: you can combine the settings on the executor and on the run config to get the desired resource utilization:
    with Flow(
            FLOW_NAME,
            storage=STORAGE,
            run_config=KubernetesRun(
                labels=["k8s"],
                cpu_request=0.5,
                memory_request="2Gi",
            ),
    ) as flow:
    b

    Ben Welsh

    7 months ago
    Ah. So the request number should go in the flow, perhaps, and not in the agent config.
    Anna Geller

    Anna Geller

    7 months ago
    correct, this way you can override whatever is set on the agent
    b

    Ben Welsh

    7 months ago
    I think that did it. Thanks for your help.