https://prefect.io logo
Title
k

Kelvin DeCosta

10/07/2022, 12:11 PM
I've gone through the
dataflow-ops
repo and I am planning to create an ECS service for a Prefect Agent that submits ECS Tasks that run the Prefect Deployments. In the repo, as per my understanding, the agent runs as an ECS based on an ECS Task Definition which requires a Docker Image. Does this Docker image need the different Python packages required by the
flows
? Or can it be lightweight and only include packages necessary for
prefect
?
šŸ’Æ 2
āœ… 1
By lightweight, I mean, can this image be built from just the
prefect-3.10
base image? I'd like to have: • 1 Docker Image with the bare minimum for a Prefect Agent • 1 Docker Image with all the dependencies and private modules (edit: for the flows)
a

Anna Geller

10/07/2022, 12:20 PM
Great question!
Or can it be lightweight and only include packages necessary for
prefect
?
This is correct, it can be totally lightweight and you can build an ECR image that needs custom packages separately and supply that image to your
ECSTask
block
šŸ‘šŸ½ 1
šŸ‘ 1
k

Kelvin DeCosta

10/07/2022, 12:21 PM
Thank you for the reply! This makes things much more nicer!
a

Anna Geller

10/07/2022, 12:21 PM
In fact, this is how the dataflow-ops repo is built -- the agent uses by default this base Prefect image https://github.com/anna-geller/dataflow-ops/blob/main/infrastructure/ecs_cluster_prefect_agent.yml#L41
k

Kelvin DeCosta

10/07/2022, 12:22 PM
Ah, I must have missed this!
a

Anna Geller

10/07/2022, 12:22 PM
no worries, great question šŸ™Œ keep us posted how it goes -- if you have questions about this repo, feel free to submit an issue there and I can respond directly
actually, you were right, right now I'm overriding the image by the one with dependencies here:
image=${{ needs.ecr-repo.outputs.image }}
this was deliberate so that you can run the flows directly in the agent container if you need to do that for low latency workflows
šŸ‘ 1
ECS Fargate tasks take some time to provision the containers so if you need to spin up something quickly (such as the maintenance flow here), this would run immediately and have all dependencies it needs tradeoffs -- definitely something you can customize based on your design decisions
k

Kelvin DeCosta

10/07/2022, 12:27 PM
Thank you, and yes I'll keep you posted!
šŸ‘ 1