https://prefect.io logo
Title
k

Krishnan Chandra

11/10/2022, 2:01 PM
I’m trying to work out how to run a specific kind of workflow in ECS. The documentation for the ECSTask infrastructure mentions:
• You must configure remote Storage. Local storage is not supported for ECS tasks. The most commonly used type of storage with
ECSTask
is S3. If you leverage that type of block, make sure that
s3fs
is installed within your agent and flow run environment. The easiest way to satisfy all the installation-related points mentioned above is to include the following commands in your Dockerfile:
Normally, I run flows that are stored within the same Docker image that contains the agent, and it looks like the
DockerContainer
and
KubernetesJob
infrastructures both support this (I primarily use
KubernetesJob
and
Process
infra atm): https://prefect-community.slack.com/archives/CL09KU1K7/p1666879201904529 It seems like it should be possible to run flows directly within the agent container on ECS: https://prefect-community.slack.com/archives/CL09KU1K7/p1665145511193949?thread_ts=1665144718.756049&cid=CL09KU1K7, but I see that the example in dataflow-ops utilizes S3 storage. If possible, I would like to just run flows within the agent container itself on ECS and specify that the code is there too - is there a way to specify for ECS that the flow code is inside the image that’s being run? If not, is this on the roadmap anywhere? Thanks for reading :)
1
As I understand it, the solution might be to run the agents within ECS, but use the
Process
infra block as shown here?
b

Bianca Hoch

11/16/2022, 5:38 PM
Hi Krishnan, to my understanding running the flows within the agent container on ECS is totally feasible. All the is really required is a correct filepath that points Prefect to where the code lives in the docker container. With that being said, you should be able to use the `Process`infrastructure block as shown in the example.
k

Krishnan Chandra

11/17/2022, 4:01 PM
Thanks Bianca! I managed to get it working a few days ago, but I really appreciate the follow-up 🙂
1
🚀 1
b

Bianca Hoch

11/17/2022, 5:52 PM
That's awesome news! Glad to hear it.