Is the prefect2 setup for AWS ECS kind of the same as it was for prefect1? I followed this tutorial once and it was super-smooth for prefect1 https://towardsdatascience.com/serverless-data-pipelines-made-easy-with-prefect-and-aws-ecs-fargate-7e25bacb450c

Indeed, that walkthrough is great. However I think Anna’s other article would be better for Orion as of now since there’s only Kubernetes flow runner currently https://towardsdatascience.com/distributed-data-pipelines-made-easy-with-aws-eks-and-prefect-106984923b30

Ah, okay – that’s unfortunate, as my k8 experiences have been dreadful. So currently for prefect2 only k8 but is ECS on the roadmap, or not at all?

Thanks, Noah 🙏 both articles are super outdated so it would be best if you could use docs for up-to-date info and avoid any confusion. Re ECS Fargate for 2.0, check this page

Ah, that’s so awesome – thanks @Anna Geller!

Incognito is your friend @Jan Domanski 😉

It’s the 20th browser window, but hey, thanks? 😛

@Noah Holm we definitely understand that but the ECS Fargate API is hard to integrate with, you can use the EKS on Fargate with

KubernetesFlowRunner

in 2.0 for now

It’s interesting looking at the community discussion you linked above Anna, my reservations around k8 are exactly the same. Too much complexity for my appetite

Could you elaborate, why do you think so? As long as you have a K8s cluster and you have some Prefect agent that simultaneously points to Prefect API via

PREFECT_API_URL

and to K8s cluster via

kubeconfig

, all you need is a

KubernetesFlowRunner

on your deployment spec and you should be good to go. I'll try to build some recipe for that in the next week and will post on Discourse and tag it under https://discourse.prefect.io/tag/kubernetes-flow-runner

Was just gonna ask you about that Anna. I have little Kubernetes experience but from what I’ve heard and read my feeling is that using Prefect on Kubernetes shouldn’t be too hard. Seems like you agree there? When you start having ingress to web servers, storage for databases or such it seems to be quite complex though. (Ofc that assumes you use Prefect Cloud to get rid of maintaining web server and db)

Seems like you agree there?

100% - it should be easy, if it's not we are doing a bad job either on the product side or on the docs/tutorial/education side - here I believe it's the latter

I think for a lot of us introducing K8 in the first place represents a massive complexity onboarding. I’m sure if you’re already using it then integrating with prefect is par for the course. A lot of us have decided to ‘skip’ the k8 abstraction. Serverless or dynamically spinning up VM is much more interesting.

I can totally understand you in that regard, managing servers is not something you should have to worry about if you are a data scientist - we are aware of the problem here and the solution we suggested in the shared discussion is to still use serverless - even still use Fargate - but exposed as a Kubernetes service. If EKS on Fargate is confusing, maybe you can check my old blog post about the topic here