hi everyone 👋 We are interesting in using ECS Agent with Prefect 2.0. Is this something already available ? Cannot find anything related to that in the Orion doc so far. cc @Julian Mehnle

I don’t think that has been implemented yet. Just the Kubernetes/Docker Flow Runners at the moment

There have been a lot of difficulties integrating with AWS’s API in 1.0 so we’re intending to implement it very carefully this time 🙂

What would you recommend for a greenfield project that'll run on AWS, with a strong hesitance to adopt Kubernetes at this time? Create your own fleet of EC2 instances that run Docker?

Unless Michael corrects me, I think that is right. Agent on EC2 with Flows that use DockerFlowRunner

@Julian Mehnle in Prefect 2.0, this is even easier than that! Given that the same agent process can simultaneously deploy flow runs as Docker containers, subprocesses, or Kubernetes jobs, depending on which

flow_runner

type you attach to your

DeploymentSpec

. I have a rough draft of a tutorial for AWS EC2 agent and was planning to polish it more before publishing, but given you need it now, I will share it already

Thanks, that's useful. When would you say that ECS (not EC2) support is likely to be available in 2.0? Some time in 2022?

Would you be open to use EKS on Fargate instead of ECS Fargate?

Open, but less likely.

From the perspective of cost and convenience benefits, Fargate is definitely attractive, but in general ECS has a really convoluted API and it's hard to build an interface that will match user's expectations since everyone seems to want to use ECS differently 😄 Some want to use it with EC2, some with ECS data plane, then there are different capacity providers, complex networking setup that many users would want Prefect to infer automatically from a default VPC, while more advanced users want to build it via a custom VPC deployed with Terraform - a really tough problem tbh. That's why we need more time to think about the right way of exposing that API to users. In contrast, if you would use EKS on Fargate, it's "just" Kubernetes in the end which has a standardized API we can support out of the box with

KubernetesFlowRunner

. And for that, we already have this awesome tutorial from Laura: You could create an EKS on Fargate cluster using eksctl in a single command (LMK if you need help with that) and then follow the tutorial from Laura to deploy the execution layer

Understood. We will likely start with a small fleet of EC2 instances running the Prefect agent. We'll cross the K8s bridge if and when that becomes necessary.