please let me know if people think this is indicating a misunderstanding of something, but I'm interested if this concept makes sense. I'd like a (large) ECS Task to run a flow with (containerized) subflows, but on that same large ECS Task. As in, I'd like the ECS worker to create a big ECS task, which will run all of the workflow / subflows, internal to that Task, (and not spawn off new ECS tasks for subflows). The reasoning is that pulling the image and starting these containers is relatively inefficient compared to running it local to the parent Task. Is this wrong headed? Is there a pattern for doing this?

Hey @Philip MacMenamin! For my own understanding, why do you want to run child flows in separate containers? If you're starting those containers from an ECS task, you'd have to wrangle with Docker in Docker, which can get hairy.

right, this was the concern. Running docker within docker is getting strange. Another option might be to minimize the child containers of each docker image to one, and have the subflows sequentially run in the appropriate container, without killing it upon completion?

Gotcha, thanks for that info! We have a feature in beta that might help with your use case. First, if possible, I recommend using a process worker, as it'll allow you to avoid any Docker-in-Docker issues. Second, via

prefect_docker

, we have a

@docker

decorator you can use to run child flows in a Docker container, and you could customize the image for each child flow run via job variables.

ok! I can read about this.

Also, here are the docs related to this: https://docs.prefect.io/v3/advanced/submit-flows-directly-to-dynamic-infrastructure. We're actively working on this, so please let me know if you have any questions or feedback.