Hi prefect community, I have a question: I have a data processing pipeline in which different processing steps require different mutually exclusive environments, which are dockerized. I am trying to implement this pipline in prefect, and I am trying to map the requirement that each dedicated computing step has to run in a different container onto the prefect logic of flows and tasks and I could use some input here. What I am currently trying out is to have each processing step being represented by a single flow with a single task, and those flows are deployed with their respective docker images. Then, those flows are logically grouped by one "super"-flow. I guess I will have to handle concurrency then at the level of deployments. Does this make sense at all, or is there a way of sending individual tasks to deployments? Or do you have different recommendation for the use case? thanks!

hi @Georg Raiser - this is a very common use case in prefect! so yes i think you're on a path that makes sense here > What I am currently trying out is to have each processing step being represented by a single flow with a single task, and those flows are deployed with their respective docker images. Then, those flows are logically grouped by one "super"-flow. I guess I will have to handle concurrency then at the level of deployments. just to say it back to you, the historical way to do this is • create deployments for each "step". deployments select a work pool that define the infra it should run on (k8s, ecs etc) • in a "parent" (or "super") flow, compose

run_deployment("my-step-flow/my-deployment")

calls (which will be treated as subflows automatically) so that you can run / deploy your "flow of flows" as needed and programatically handle failures on the other infras but this is also a big reason why we're exploring ad-hoc infra submission lately! so you may be interested to check this out (cc @alex)

I think that your use case would work well with the

@docker

decorator that was recently introduced to

prefect-docker

! There's some info about it in the docs here, but feel free to reach out with any questions you have while trying it out!

Thanks a lot @Nate and @alex, this indeed looks very promising. I'll play around with it!