https://prefect.io logo
Title
m

Matt Melgard

08/30/2022, 9:18 PM
Hey all, so I’m using the Kubernetes Agent to run flows and I was wondering what if any support there is for defining some kind of DAG like workflow where parallel tasks can run in separate pods? Is that something that Prefect can do with the kube agent? Feel free to RTFM me if I missed something critical in the docs, just not finding anything myself
1
Is this handled by subflows by chance?
n

Nate

08/30/2022, 9:26 PM
Hey @Matt Melgard! Depending on what you're up to, the following might work for you: You could create 2 deployments (say,
orchestrator
and
worker
) and create the worker deployment with an
-ib kubernetes-job/your-k8s-job-block
as needed then in the
orchestrator
you could use the client to `create_flow_run_from_deployment` , where you'd pass the
deployment_id
of the worker and the
params
it needs to run with something like:
from prefect.utilities.asyncutils import sync_compatible

@task
@sync_compatible
async def submit_worker_flows(deployment_id: str, params: Dict) -> FlowRun:
    """Async task to create a flow run from the `worker` deployment
    Args:
        deployment_id (str): Prefect Deployment ID of `modeling_worker` flow deployment
        params (Dict): Everything an instance of `worker` needs to run
    """
    async with get_client() as client:
        await client.create_flow_run_from_deployment(
            deployment_id=deployment_id, parameters=params
        )
that way, the
worker
flows can run on their own infra and you could poll for state from the
orchestrator
as needed
👍 1
"subflows" will run on their parent's infra
m

Matt Melgard

08/30/2022, 9:59 PM
So, if I’m understanding correctly then, we’re kind of having to construct the orchestration logic to run separate tasks on separate pods ourselves rather than tasks/flows themselves supporting that through their API’s directly?
How does this kind of thing show up in the UI? Would deployments triggered this way show up as “children” of the parent flow in some way?
n

Nate

08/30/2022, 11:00 PM
not currently - these flow runs wouldn't be directly associated with the parent you invoked them from however you raise a good point, and we're in the process of building out our support for subflows on their own infra to allow for better visibility for the time being you may find client.read_flow_runs helpful in monitoring the state of flows you kick off with the client