onlycasual1
11/18/2025, 3:31 PMMarvin
11/18/2025, 3:31 PMMarvin
11/18/2025, 3:34 PMonlycasual1
11/18/2025, 3:40 PMMarvin
11/18/2025, 3:40 PMMarvin
11/18/2025, 3:45 PMfrom prefect import flow, task
from prefect.task_runners import ThreadPoolTaskRunner
from prefect.deployments import run_deployment
from prefect.flow_runs import wait_for_flow_run
# tiny task to launch a child flow run and return its id
@task
def launch_child(deployment_name: str, params: dict) -> str:
# as_subflow=True links runs in the UI; each run will still execute in its own pod
fr = run_deployment(
name=deployment_name,
parameters=params,
as_subflow=True,
timeout=0, # do not wait here
# work_queue_name="k8s-queue", # optional routing
# job_variables={"cpu": "1", "memory": "2Gi"} # optional pod overrides
)
return str(fr.id)
# tiny task to wait on a child run to finish
@task
def wait_child(flow_run_id: str):
return wait_for_flow_run(flow_run_id).state
@flow(task_runner=ThreadPoolTaskRunner(max_workers=32))
def parent(items: list[int]):
# 1) fan out: trigger N child flows (N pods)
launches = [launch_child.submit("child-flow/my-deployment", {"item": i}) for i in items]
child_run_ids = [f.result() for f in launches]
# 2) wait for all children concurrently (optional)
waits = [wait_child.submit(fr_id) for fr_id in child_run_ids]
results = [w.result() for w in waits]
return results
Why this works for 1 pod per unit
- Each run_deployment call creates a new flow run routed to your Kubernetes work pool; the worker spins a new Job/Pod for it
- The parent remains in its own pod; the child runs each get their own pod
- Using small @task wrappers lets you launch and wait concurrently in a sync flow
Notes and tips
- Ensure the child flow has a deployment and is routed to your Kubernetes work pool
- Tune concurrency in the parent with ThreadPoolTaskRunner(max_workers=...) to match your desired fan-out
- Make sure your Kubernetes work pool/worker can run that many runs at once (work pool concurrency, cluster autoscaling, image pull time, etc.)
- Use job_variables in run_deployment to tweak per-pod resources, env, image, etc.
- Use idempotency_key if you want to dedupe launches on retries
Docs
- Run work concurrently
- Task runners
- Trigger deployments (run_deployment)
If you share how many concurrent “units” you need and your work pool name, I can tailor the max_workers, work queue, and job_variables example for your setup.