https://prefect.io
Join Slack
I've gone through the `dataflow-ops` repo and I am planning to create an ECS service for a Prefect A...
# ask-community
k
I've gone through the 
dataflow-ops
repo and I am planning to create an ECS service for a Prefect Agent that submits ECS Tasks that run the Prefect Deployments. In the repo, as per my understanding, the agent runs as an ECS based on an ECS Task Definition which requires a Docker Image. Does this Docker image need the different Python packages required by the 
flows
? Or can it be lightweight and only include packages necessary for 
prefect
?
šŸ’Æ 2
āœ… 1
By lightweight, I mean, can this image be built from just the 
prefect-3.10
base image? I'd like to have: ā€¢ 1 Docker Image with the bare minimum for a Prefect Agent ā€¢ 1 Docker Image with all the dependencies and private modules (edit: for the flows)
a
Great question!
Or can it be lightweight and only include packages necessary for 
prefect
?
This is correct, it can be totally lightweight and you can build an ECR image that needs custom packages separately and supply that image to your 
ECSTask
block
šŸ‘šŸ½ 1
šŸ‘ 1
k
Thank you for the reply! This makes things much more nicer!
a
In fact, this is how the dataflow-ops repo is built -- the agent uses by default this base Prefect image https://github.com/anna-geller/dataflow-ops/blob/main/infrastructure/ecs_cluster_prefect_agent.yml#L41
k
Ah, I must have missed this!
a
no worries, great question šŸ™Œ keep us posted how it goes -- if you have questions about this repo, feel free to submit an issue there and I can respond directly
actually, you were right, right now I'm overriding the image by the one with dependencies here:
Copy code
image=${{ needs.ecr-repo.outputs.image }}
this was deliberate so that you can run the flows directly in the agent container if you need to do that for low latency workflows
šŸ‘ 1
ECS Fargate tasks take some time to provision the containers so if you need to spin up something quickly (such as the maintenance flow here), this would run immediately and have all dependencies it needs tradeoffs -- definitely something you can customize based on your design decisions
k
Thank you, and yes I'll keep you posted!
šŸ‘ 1
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by