I ve gone through the `dataflow ops` repo and I am planning Prefect Community #ask-community

I've gone through the `dataflow-ops` repo and I am...

Kelvin DeCosta

10/07/2022, 12:11 PM

I've gone through the

dataflow-ops

repo and I am planning to create an ECS service for a Prefect Agent that submits ECS Tasks that run the Prefect Deployments. In the repo, as per my understanding, the agent runs as an ECS based on an ECS Task Definition which requires a Docker Image. Does this Docker image need the different Python packages required by the

flows

? Or can it be lightweight and only include packages necessary for

prefect

💯 2

✅ 1

Kelvin DeCosta

10/07/2022, 12:15 PM

By lightweight, I mean, can this image be built from just the

prefect-3.10

base image? I'd like to have: • 1 Docker Image with the bare minimum for a Prefect Agent • 1 Docker Image with all the dependencies and private modules (edit: for the flows)

Anna Geller

10/07/2022, 12:20 PM

Great question!

Or can it be lightweight and only include packages necessary for
prefect
?

This is correct, it can be totally lightweight and you can build an ECR image that needs custom packages separately and supply that image to your

ECSTask

block

👍🏽 1

👍 1

Kelvin DeCosta

10/07/2022, 12:21 PM

Thank you for the reply! This makes things much more nicer!

Anna Geller

10/07/2022, 12:21 PM

In fact, this is how the dataflow-ops repo is built -- the agent uses by default this base Prefect image https://github.com/anna-geller/dataflow-ops/blob/main/infrastructure/ecs_cluster_prefect_agent.yml#L41

Kelvin DeCosta

10/07/2022, 12:22 PM

Ah, I must have missed this!

Anna Geller

10/07/2022, 12:22 PM

no worries, great question 🙌 keep us posted how it goes -- if you have questions about this repo, feel free to submit an issue there and I can respond directly

Anna Geller

10/07/2022, 12:25 PM

actually, you were right, right now I'm overriding the image by the one with dependencies here:

Copy code

image=${{ needs.ecr-repo.outputs.image }}

this was deliberate so that you can run the flows directly in the agent container if you need to do that for low latency workflows

👍 1

Anna Geller

10/07/2022, 12:27 PM

ECS Fargate tasks take some time to provision the containers so if you need to spin up something quickly (such as the maintenance flow here), this would run immediately and have all dependencies it needs tradeoffs -- definitely something you can customize based on your design decisions

Kelvin DeCosta

10/07/2022, 12:27 PM

Thank you, and yes I'll keep you posted!

👍 1

Open in Slack

Previous Next