Paul Crook
11/10/2025, 2:22 AMMarvin
11/10/2025, 2:22 AMMarvin
11/10/2025, 2:29 AMrun_deployment(...). This keeps everything visible in the UI as a subflow tree and gives you clean separation of infrastructure.
Example:
from prefect import flow, task
from prefect.deployments import run_deployment
@flow
def parent_flow(params):
# Trigger a GPU subflow deployment
gpu_run = run_deployment(
name="gpu-flow/gpu-deploy",
parameters={"x": params["x"]},
# Optional: override job variables at runtime (pool-specific)
# job_variables={"resources": {"limits": {"<http://nvidia.com/gpu|nvidia.com/gpu>": 1}}}
)
# Trigger a CPU subflow deployment
cpu_run = run_deployment(
name="cpu-flow/cpu-deploy",
parameters={"y": params["y"]}
)
return {"gpu_state": gpu_run.state, "cpu_state": cpu_run.state}
- run_deployment waits for the triggered deployment to complete before returning and shows up as a subflow in the UI by default. Signature for reference:
prefect.deployments.flow_runs.run_deployment(
name, client=None, parameters=None, scheduled_time=None,
flow_run_name=None, timeout=None, poll_interval=5, tags=None,
idempotency_key=None, work_queue_name=None, as_subflow=True, job_variables=None
) -> FlowRun
- Docs: Run deployments, Deployments
Tip: If you want to kick off multiple subflows in parallel, wrap run_deployment in a Prefect task and submit them concurrently from the parent flow:
from prefect import flow, task
from prefect.deployments import run_deployment
@task
def trigger(name, parameters=None):
return run_deployment(name=name, parameters=parameters)
@flow
def orchestrator():
a = trigger.submit("gpu-flow/gpu-deploy", {"x": 1})
b = trigger.submit("cpu-flow/cpu-deploy", {"y": 2})
return a.result(), b.result()
2) Use background tasks served by task workers (per-task infrastructure, not work pools)
- If your real goal is to run “this specific function on a different machine/host,” background tasks can help.
- Start one or more task workers with prefect task serve ... on the machines you want to run the work.
- From your flow, call tasks with my_task.delay(...). These runs are picked up by task workers and can execute on different machines than the parent flow.
- Note: Background tasks do not use work pools; you manage where they run by where you start the task workers.
Example:
from prefect import flow, task
@task
def heavy_gpu_task(data):
# Runs wherever a task worker is serving this task (e.g., a GPU box)
return f"processed {data}"
@flow
def parent_flow():
fut = heavy_gpu_task.delay(data="payload")
return fut.result() # wait for the background task
- CLI to serve tasks: prefect task serve ./path/to/file.py:heavy_gpu_task
- Docs/blog: Background tasks, Why background tasks matter and example repo: prefect-background-task-examples
Choosing a pattern
- If you specifically need different work pools (e.g., Docker vs Kubernetes, or different pool-level templates), use the subflow + deployment approach with run_deployment.
- If you need to run certain functions on different machines but don’t need work pools, task workers with .delay() are lighter-weight.Marvin
11/10/2025, 2:29 AMjob_variables for your work pool type.Paul Crook
11/10/2025, 2:35 AMPaul Crook
11/10/2025, 2:36 AM