Hello Prefect Community I would like to understand how other Prefect Community #ask-community

Hello Prefect Community I would like to understan...

Andrew Allen

08/07/2025, 5:55 PM

Hello Prefect Community I would like to understand how others are sub-flows in their own K8s pod without using

RayTaskRunner

DaskTaskRunner

on kubernetes.

Andrew Allen

08/07/2025, 5:55 PM

For example

Copy code

┌─────────────────────────────────────────────────────────────────┐
│                     Kubernetes Cluster                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─────────────────┐              ┌─────────────────────────┐    │
│  │ Prefect Server  │◄─────────────┤    Prefect Agent        │    │
│  │ - Flow Registry │              │ - Polls for work       │    │
│  │ - Orchestration │              │ - Creates K8s Jobs     │    │
│  │ - State Mgmt    │              │ - Monitors execution   │    │
│  └─────────────────┘              └─────────────────────────┘    │
│           │                                   │                  │
│           │                                   │                  │
│           ▼                                   ▼                  │
│  ┌─────────────────┐              ┌─────────────────────────┐    │
│  │   Main Flow     │              │   K8s Job Controller    │    │
│  │     Pod         │◄─────────────┤ - Pod lifecycle mgmt   │    │
│  │ - Coordinates   │              │ - Resource allocation  │    │
│  │ - Defines deps  │              │ - Failure handling     │    │
│  └─────────────────┘              └─────────────────────────┘    │
│           │                                   │                  │
│           │ Spawns Sub-Flows                  │                  │
│           ├───────────────┬───────────────────┼──────────────────┤
│           │               │                   │                  │
│           ▼               ▼                   ▼                  │
│  ┌─────────────┐ ┌─────────────┐    ┌─────────────┐             │
│  │ Sub-Flow 1  │ │ Sub-Flow 2  │    │ Sub-Flow 3  │             │
│  │    Pod      │ │    Pod      │    │    Pod      │             │
│  │             │ │             │    │             │             │
│  │ ┌─────────┐ │ │ ┌─────────┐ │    │ ┌─────────┐ │             │
│  │ │ Task 1A │ │ │ │ Task 2A │ │    │ │ Task 3A │ │             │
│  │ └─────────┘ │ │ └─────────┘ │    │ └─────────┘ │             │
│  │ ┌─────────┐ │ │ ┌─────────┐ │    │ ┌─────────┐ │             │
│  │ │ Task 1B │ │ │ │ Task 2B │ │    │ │ Task 3B │ │             │
│  │ └─────────┘ │ │ └─────────┘ │    │ └─────────┘ │             │
│  └─────────────┘ └─────────────┘    └─────────────┘             │
│           │               │                   │                  │
│           └───────────────┼───────────────────┘                  │
│                           │                                      │
│                           ▼                                      │
│                  Reports State Back                              │
└─────────────────────────────────────────────────────────────────┘

Brendan Dalpe

08/11/2025, 1:29 PM

@Andrew Allen you'll most likely need to look at using

asyncio.gather

to run your subflows if you need your main flow to wait for the subflows to complete. Take a look at a simple example here: https://github.com/PrefectHQ/prefect-recipes/blob/main/flows-starter/async_subflows.py

Brendan Dalpe

08/11/2025, 1:30 PM

If you don't need to wait, you can just call

run_deployment

to dispatch the subflow in a separate Pod: https://docs.prefect.io/v3/how-to-guides/deployments/run-deployments#run-a-deployment-from-python

Andrew Allen

08/11/2025, 3:54 PM

Hi @Brendan Dalpe, thank you for this info. Will the

asyncio.gather

method invoke a K8s job for each

sub-flow

Brendan Dalpe

08/11/2025, 3:54 PM

Yes

2 Views

Open in Slack

Previous Next