<@ULVA73B9P> How to run different tasks in the sam...
# ask-marvin
p
@Marvin How to run different tasks in the same flow in different workpools?
m
thought for 370.8 seconds
Short answer: not directly. In Prefect 3.x, a flow run executes on the work pool configured for its deployment, and all tasks in that flow run share that same infrastructure. Common patterns to achieve “different infra per unit of work”: 1) Split into subflows and deploy each to its own work pool - Put the GPU/CPU/etc. work into separate flows. - Create separate deployments for each subflow, each pointing at the appropriate work pool. - From the parent flow, call those deployments with
run_deployment(...)
. This keeps everything visible in the UI as a subflow tree and gives you clean separation of infrastructure. Example:
Copy code
from prefect import flow, task
from prefect.deployments import run_deployment

@flow
def parent_flow(params):
    # Trigger a GPU subflow deployment
    gpu_run = run_deployment(
        name="gpu-flow/gpu-deploy",
        parameters={"x": params["x"]},
        # Optional: override job variables at runtime (pool-specific)
        # job_variables={"resources": {"limits": {"<http://nvidia.com/gpu|nvidia.com/gpu>": 1}}}
    )

    # Trigger a CPU subflow deployment
    cpu_run = run_deployment(
        name="cpu-flow/cpu-deploy",
        parameters={"y": params["y"]}
    )

    return {"gpu_state": gpu_run.state, "cpu_state": cpu_run.state}
-
run_deployment
waits for the triggered deployment to complete before returning and shows up as a subflow in the UI by default. Signature for reference:
Copy code
prefect.deployments.flow_runs.run_deployment(
  name, client=None, parameters=None, scheduled_time=None,
  flow_run_name=None, timeout=None, poll_interval=5, tags=None,
  idempotency_key=None, work_queue_name=None, as_subflow=True, job_variables=None
) -> FlowRun
- Docs: Run deployments, Deployments Tip: If you want to kick off multiple subflows in parallel, wrap
run_deployment
in a Prefect task and submit them concurrently from the parent flow:
Copy code
from prefect import flow, task
from prefect.deployments import run_deployment

@task
def trigger(name, parameters=None):
    return run_deployment(name=name, parameters=parameters)

@flow
def orchestrator():
    a = trigger.submit("gpu-flow/gpu-deploy", {"x": 1})
    b = trigger.submit("cpu-flow/cpu-deploy", {"y": 2})
    return a.result(), b.result()
2) Use background tasks served by task workers (per-task infrastructure, not work pools) - If your real goal is to run “this specific function on a different machine/host,” background tasks can help. - Start one or more task workers with
prefect task serve ...
on the machines you want to run the work. - From your flow, call tasks with
my_task.delay(...)
. These runs are picked up by task workers and can execute on different machines than the parent flow. - Note: Background tasks do not use work pools; you manage where they run by where you start the task workers. Example:
Copy code
from prefect import flow, task

@task
def heavy_gpu_task(data):
    # Runs wherever a task worker is serving this task (e.g., a GPU box)
    return f"processed {data}"

@flow
def parent_flow():
    fut = heavy_gpu_task.delay(data="payload")
    return fut.result()  # wait for the background task
- CLI to serve tasks:
prefect task serve ./path/to/file.py:heavy_gpu_task
- Docs/blog: Background tasks, Why background tasks matter and example repo: prefect-background-task-examples Choosing a pattern - If you specifically need different work pools (e.g., Docker vs Kubernetes, or different pool-level templates), use the subflow + deployment approach with
run_deployment
. - If you need to run certain functions on different machines but don’t need work pools, task workers with
.delay()
are lighter-weight.
If you can share your use case (e.g., CPU vs GPU split, different clusters, cost isolation), I can suggest the cleanest implementation and provide tailored examples, including how to set
job_variables
for your work pool type.
p
Thanks. My specific case is that one group of tasks need to run in a specific Docker container, while proceeding and following tasks can run in a general process environment on the same machine.
Sounds like option (1), sub-flows would work in this case.