hi everyone :wave: We are interesting in using EC...
# prefect-community
hi everyone ๐Ÿ‘‹ We are interesting in using ECS Agent with Prefect 2.0. Is this something already available ? Cannot find anything related to that in the Orion doc so far. cc @Julian Mehnle
I donโ€™t think that has been implemented yet. Just the Kubernetes/Docker Flow Runners at the moment
There have been a lot of difficulties integrating with AWSโ€™s API in 1.0 so weโ€™re intending to implement it very carefully this time ๐Ÿ™‚
๐Ÿ‘ 2
๐Ÿ’ฏ 1
What would you recommend for a greenfield project that'll run on AWS, with a strong hesitance to adopt Kubernetes at this time? Create your own fleet of EC2 instances that run Docker?
(I'm working with @Matt Delacour on this.)
Unless Michael corrects me, I think that is right. Agent on EC2 with Flows that use DockerFlowRunner
@Julian Mehnle in Prefect 2.0, this is even easier than that! Given that the same agent process can simultaneously deploy flow runs as Docker containers, subprocesses, or Kubernetes jobs, depending on which
type you attach to your
. I have a rough draft of a tutorial for AWS EC2 agent and was planning to polish it more before publishing, but given you need it now, I will share it already
Thanks, that's useful. When would you say that ECS (not EC2) support is likely to be available in 2.0? Some time in 2022?
Would you be open to use EKS on Fargate instead of ECS Fargate?
Open, but less likely.
K8s introduces a whole other virtualization layer on top of a virtual data center that we're skeptical of dealing with for the foreseeable future.
FWIW, we can absolutely run a small EC2 instance fleet for now.
๐Ÿ™Œ 1
From the perspective of cost and convenience benefits, Fargate is definitely attractive, but in general ECS has a really convoluted API and it's hard to build an interface that will match user's expectations since everyone seems to want to use ECS differently ๐Ÿ˜„ Some want to use it with EC2, some with ECS data plane, then there are different capacity providers, complex networking setup that many users would want Prefect to infer automatically from a default VPC, while more advanced users want to build it via a custom VPC deployed with Terraform - a really tough problem tbh. That's why we need more time to think about the right way of exposing that API to users. In contrast, if you would use EKS on Fargate, it's "just" Kubernetes in the end which has a standardized API we can support out of the box with
. And for that, we already have this awesome tutorial from Laura:


You could create an EKS on Fargate cluster using eksctl in a single command (LMK if you need help with that) and then follow the tutorial from Laura to deploy the execution layer
Understood. We will likely start with a small fleet of EC2 instances running the Prefect agent. We'll cross the K8s bridge if and when that becomes necessary.
๐Ÿ‘ 1
(And, yes, we're using Terraform to manage the underlying infrastructure.)
๐Ÿ‘ 1