Krishnan Chandra
11/10/2022, 2:01 PM• You must configure remote Storage. Local storage is not supported for ECS tasks. The most commonly used type of storage withNormally, I run flows that are stored within the same Docker image that contains the agent, and it looks like theis S3. If you leverage that type of block, make sure thatECSTask
is installed within your agent and flow run environment. The easiest way to satisfy all the installation-related points mentioned above is to include the following commands in your Dockerfile:s3fs
DockerContainer
and KubernetesJob
infrastructures both support this (I primarily use KubernetesJob
and Process
infra atm): https://prefect-community.slack.com/archives/CL09KU1K7/p1666879201904529
It seems like it should be possible to run flows directly within the agent container on ECS: https://prefect-community.slack.com/archives/CL09KU1K7/p1665145511193949?thread_ts=1665144718.756049&cid=CL09KU1K7, but I see that the example in dataflow-ops utilizes S3 storage.
If possible, I would like to just run flows within the agent container itself on ECS and specify that the code is there too - is there a way to specify for ECS that the flow code is inside the image that’s being run? If not, is this on the roadmap anywhere?
Thanks for reading :)Process
infra block as shown here?Bianca Hoch
11/16/2022, 5:38 PMKrishnan Chandra
11/17/2022, 4:01 PMBianca Hoch
11/17/2022, 5:52 PM