Hey folks! We’ve been trying to wrap our heads around getting a Fargate Flow to call/start another docker container that will also run on fargate, but if we use the normal Prefect DockerTasks we hit the always-hilarious-but-not-really docker in docker problem. It’s just occurred to me we may have been doing this completely ass ways. Am I right in thinking that if I want the following: • Container A contains Prefect Flow • Container B contains some business logic code that knows nothing about the Prefect Flow • Fargate Task that allows mounting of a volume between A and B Then I should be able to create both of these in the containerDefinitions of the FargateTaskEnvironment? It would mean we’re properly able to get the side-by-side/peer container approach working, rather than the nested/d-in-d approach that the DockerTasks would give us… And, far more importantly, are there examples of how to do the above??? 😄

Does the flow running in Container A need to be able to talk to Container B?

Yes - A needs to be able to trigger 2 commands [currently just shell scripts] in B

So I guess there are two ways you could do this: 1. Use the FargateTaskEnvironment with a containerDefinitions that lists two containers (A and B) that are somehow mounted for communication. 2. Refactor your flow to where instead of running a command on a container have the flow run external Fargate tasks (using boto3) that trigger the shell scripts using whichever image you want to run it on. FWIW I have not attempted what you are trying so I don’t have a whole lot of guidance on how to accomplish it 🙂

That’s fair enough, what I’m mostly looking for right now is confirmation that the A and B containers can be created in

FargateTaskEnvironment.containerDefinitions

And I think I’ve figured out a reasonably dastardley way of accomplishing the comms as well - if I use 2 mounts, one to get a data mount from A to B, and then a second to get a script/commands mount from B to A, then I think I can get away with it… Do you know are there any examples in the docs anywhere on more than 1 container being instantiated in the

FargateTaskEnvironment

?? I’m also assuming that I then run these flows using Docker agent rather than Fargate, as the flow will be providing it’s own Fargate env?

No examples that I am aware of but the

conatinerDefinitions

takes a list of dicts so it should be straightforward. There is an example using the KubernetesJobEnvironment with two containers but it’s pretty different of course due to being k8s. You could use to Docker agent if you want to run the initial prefect job as a docker container on a machine or using Fargate agent it also an option (it would just launch the init job as a fargate task) so either would wor.

Great, thanks for your help @josh!! I’ll let you know if it works