Hi all, I am trying to run a sub-flow on a different docker image than my main flow and pass data back from my sub-flow to my main flow using

prefect 2.0b7-python3.8

. I'm trying to orchestrate the images through

DeploymentSpecs

with

DockerFlowRunners

as the flow_runners. Currently what happens is the sub flow runs on the same image as the main flow unless the sub flow is deployed independently. Would greatly appreciate any ideas or insights into current feasibility!

passing data between containers doesn't work natively since those run in entirely different processes, you would need to either use subflows instead or persist the data manually yourself + leverage the orchestrator pattern https://discourse.prefect.io/t/how-to-create-a-flow-run-from-deployment-orchestrator-pattern/803

Hi @Anna Geller, thanks for the response! I think I understand how to do it by "persist the data manually yourself + leverage the orchestrator pattern", but I don't quite understand the "use subflows instead". I thought my current approach was trying to use subflows but maybe I misunderstood.

I think Anna means you need to start a new flow run from a deployment like this

subflows pattern means running your flows directly from the main/parent flow orchestrator pattern means triggering your child flow runs via an API call which can be triggered e.g. using

client.create_flow_run_from_deployment(depl_id)