https://prefect.io logo
Title
n

Nate

03/02/2023, 8:00 PM
@Marvin how can i run flows in docker containers on ECS using prefect 2?
m

Marvin

03/02/2023, 8:01 PM
Ah, running flows with Prefect in Docker containers on ECS. That sounds quite complicated! I can definitely help you out here. To successfully run flows with Prefect in Docker containers on ECS, you'll need to install the necessary packages by adding this command to your Dockerfile:
FROM prefecthq/prefect:2-python3.9 # example base image 
RUN pip install s3fs prefect-aws
You should also allocate enough CPU and memory to your agent, as well as add environment variables to your agent environment such as
AWS_MAX_ATTEMPTS
and
AWS_RETRY_MODE
, as these will enable Prefect to better access your ECS resources. Setting a work-queue concurrency limit is also recommended. When creating a deployment, you should specify the
process
infrastructure type and configure remote Storage (local storage is not supported for ECS tasks). For more details about this, check out the [Flow deployments - Prefect 2 - Coordinating the world's dataflows](https://docs.prefect.io/tutorials/deployments/) and [Infrastructure - Prefect 2 - Coordinating the world's dataflows](https://docs.prefect.io/concepts/infrastructure/) pages. I hope this helps!
n

Nate

03/02/2023, 8:05 PM
corrections: instead of process infrastructure, you’d want to use an ECSTask infra block and you’d set resources there, not where the agent runs