Hi! I have a question regarding running prefect on AWS ECS. I am currently using Fargate to launch my flows. I have a pretty big docker image (~2-3GB uncompressed) which adds some dependencies (not only python) on top of the prefect docker image. The problem is, that for every flow Fargate pulls the image from AWS ECR (in the same VPC) which results in multiple minutes to start. Most of the runs are small, so I need to wait couple of minute to start and then they finish within a few seconds. Let’s assume I start 100 flows a day, this would result in 200-300gb of pulling the same image. My first idea was to split the image into multiple images and use subflows. Then every subflow could specify which image and dependencies it needs. Or I could try to reduce the image somehow. But in both cases even at 0.5GB per image it would result in pulling 50GB a day. I found this AWS issue regarding caching the image: https://github.com/aws/containers-roadmap/issues/696 Unfortunately caching is currently only supported for EC2 but not for Fargate. So my second idea was to use EC2 instead? But I am not sure how well it scales. This would result in startup and shutdown of EC2 instances depending on how many flows are running. So it might just shifts the startup problem as flows might need to wait for another EC2 instance to start. I used this tutorial to set-up everything for Fargate: https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df (thanks to @Anna Geller, this works great!) But I could not figure out how to do it for EC2 properly. If I understand correctly, EC2 has one IP per instance while Fargate has one IP per every flow, so the set-up would be a little different. My main problem is the long startup time of multiple minutes and I am not sure what’s the best way to deal with it. Maybe someone experienced the same problem and found a better solution?

With such a large image I’d start off with trying to optimising the image. Have you spent any time on that? E.g., looking at multi stage builds

Thanks for your reply! It is not fully optimised yet, so this would definitely help a little. But my dependencies may grow in future so might be a good idea to split them into multiple images

I think it’s very valuable to have some tricks to optimise the size so def worth looking into. Depending on what you’re including in the image it could be easy or hard ofc 🙂 But splitting them up might be relevant as well, maybe you end up making it more understandable with smaller sized flows?

There are a couple of different problems here and things to unpack so I'll try to approach it in a structured way. #1 Docker image size There are multiple ways to reduce the Docker image size. You can check e.g. this tutorial to learn more. Unfortunately using serverless the image must be pulled every time since each time your ECS task may get deployed to a completely different EC2 instance. Some questions: 1.1 Why is this image so big? Can you share the Dockerfile? 1.2 Why do you have non-Python dependencies directly in your flow Docker image? Do you trigger those non-Python processes via subprocess e.g. using

ShellTask

? If so, you could consider separating those into a separate ECS task to separate out those dependencies and potentially mitigate the issue, especially if you run flows with this non-python dependency infrequently. You could e.g. use this unofficial ecsruntask and here is a usage example for this task in a flow: ecsruntask_usage.py #2 Latency between serverless and non-serverless ECS data plane The fact that you need to wait a couple of minutes until your ECS container starts is completely normal. You need to consider all the work that AWS is doing here: • checking your workloads requirements • identify an EC2 instance that satisfies those requirements (enough CPU/memory, the right region and AZ) • pulling the image for your container • starting the container • sending logs to CloudWatch • and then Prefect needs to also do lot of work to communicate with this container to manage the task and flow run states, pull the logs, etc. If this latency is not acceptable by your workflows, you should consider either: • switching to ECS Fargate with self-managed EC2 data plane with instances being always on → you’re spot on that it doesn’t scale that well since you would need to manage that compute • switching to a Kubernetes agent on AWS EKS with managed node groups which gives almost all the same benefits of a fully managed service as Fargate does but without the “serverless latency” (but at a slightly higher price) → regarding your worry of scale, EKS makes it incredibly easy to add additional managed nodes to your cluster as you need, or you could even combine it with EKS on Fargate - I discussed this in detail in this blog post. #3 Consider AWS EKS with managed node groups instead of ECS Fargate for latency-sensitive workloads If you want to spin up an EKS cluster instead, you can do that using eksctl. To spin up a cluster with a single node you need a single command which under the hood triggers a CloudFormation stack handling all the infrastructure details:

eksctl create cluster --name=prefect-eks --nodes=1

This blog post discusses the topic in much more detail, incl. a full walkthrough and even a CI/CD pipeline for that. An additional benefit of using AWS EKS with managed node groups is that your instances are always on, therefore you don’t have to pull the Docker image if it already exists on the machine! You can do that by setting the

imagePullPolicy

on your Kubernetes job template (see example in

src/prefect/agent/kubernetes/job_template.yaml

) that you can pass to a

KubernetesRun

, or you can set up directly when starting the agent:

prefect agent kubernetes install --key YOUR_API_KEY --label eks

This will return a Kubernetes manifest for the

KubernetesAgent

that contains `imagePullPolicy`:

...
image: prefecthq/prefect:1.0.0-python3.7
imagePullPolicy: Always
...

You can see that by default it’s set to “Always”, but you can change it to

IfNotPresent

image: prefecthq/prefect:1.0.0-python3.7
imagePullPolicy: IfNotPresent

You could also set it directly on your `KubernetesRun`:

flow.run_config = KubernetesRun(image_pull_policy="imagePullPolicy")

More on that policy here. #Conclusion So overall, I totally understand your concerns and your frustration with ECS Fargate latency. Out of curiosity I once benchmarked EKS on Fargate vs. EKS on managed node groups and Fargate was 12 times slower than managed node groups. So serverless is great and all but you need to be patient with it :) Perhaps you can do a small PoC with the setup from the dbt part 2 article and compare it with the ECS Fargate setup and decide which one works better for you.

Wow I really appreciate your effort! @Anna Geller 1.1: I have some apt and one npm package installed which I use over the CLI. I will try to optimise and clean up my Dockerfile. 1.2: Yes you are right, I use ShellTasks to execute CLI commands. They are used to process/convert files before I can work on them in Python/Prefect. Those could be outsourced. I will have a look at your ECSRunTask solution. I also have one non-pip python package which is a wrapper around a custom build C++ library. Those dependencies are used for different kind of flows. Currently it is one big image with all needed dependencies. I like the idea of Kubernetes and will try if it’s worth the costs, seems pretty cool and easy to setup. Thanks for your help :)