Thread
#prefect-community
    Jan Domanski

    Jan Domanski

    4 months ago
    Is the prefect2 setup for AWS ECS kind of the same as it was for prefect1? I followed this tutorial once and it was super-smooth for prefect1 https://towardsdatascience.com/serverless-data-pipelines-made-easy-with-prefect-and-aws-ecs-fargate-7e25bacb450c
    Noah Holm

    Noah Holm

    4 months ago
    Indeed, that walkthrough is great. However I think Anna’s other article would be better for Orion as of now since there’s only Kubernetes flow runner currentlyhttps://towardsdatascience.com/distributed-data-pipelines-made-easy-with-aws-eks-and-prefect-106984923b30
    Jan Domanski

    Jan Domanski

    4 months ago
    Ah, okay – that’s unfortunate, as my k8 experiences have been dreadful. So currently for prefect2 only k8 but is ECS on the roadmap, or not at all?
    Also, towardsdatascience is now bugging me to update – it’d be so nice to have this outside the paywall ec
    Anna Geller

    Anna Geller

    4 months ago
    Thanks, Noah 🙏 both articles are super outdated so it would be best if you could use docs for up-to-date info and avoid any confusion. Re ECS Fargate for 2.0, check this page
    Jan Domanski

    Jan Domanski

    4 months ago
    Ah, that’s so awesome – thanks @Anna Geller!
    Noah Holm

    Noah Holm

    4 months ago
    Incognito is your friend @Jan Domanski 😉
    That article is helpful for me to 😄 We are using ECS Fargate atm and thinking a bit how to handle this with 2.0 in the future
    Jan Domanski

    Jan Domanski

    4 months ago
    It’s the 20th browser window, but hey, thanks? 😛
    Anna Geller

    Anna Geller

    4 months ago
    @Noah Holm we definitely understand that but the ECS Fargate API is hard to integrate with, you can use the EKS on Fargate with
    KubernetesFlowRunner
    in 2.0 for now
    Jan Domanski

    Jan Domanski

    4 months ago
    It’s interesting looking at the community discussion you linked above Anna, my reservations around k8 are exactly the same. Too much complexity for my appetite
    Anna Geller

    Anna Geller

    4 months ago
    Could you elaborate, why do you think so? As long as you have a K8s cluster and you have some Prefect agent that simultaneously points to Prefect API via
    PREFECT_API_URL
    and to K8s cluster via
    kubeconfig
    , all you need is a
    KubernetesFlowRunner
    on your deployment spec and you should be good to go. I'll try to build some recipe for that in the next week and will post on Discourse and tag it under https://discourse.prefect.io/tag/kubernetes-flow-runner
    Noah Holm

    Noah Holm

    4 months ago
    Was just gonna ask you about that Anna. I have little Kubernetes experience but from what I’ve heard and read my feeling is that using Prefect on Kubernetes shouldn’t be too hard. Seems like you agree there? When you start having ingress to web servers, storage for databases or such it seems to be quite complex though. (Ofc that assumes you use Prefect Cloud to get rid of maintaining web server and db)
    Anna Geller

    Anna Geller

    4 months ago
    Seems like you agree there?
    100% - it should be easy, if it's not we are doing a bad job either on the product side or on the docs/tutorial/education side - here I believe it's the latter
    b

    Bob Colner

    4 months ago
    I think for a lot of us introducing K8 in the first place represents a massive complexity onboarding. I’m sure if you’re already using it then integrating with prefect is par for the course. A lot of us have decided to ‘skip’ the k8 abstraction. Serverless or dynamically spinning up VM is much more interesting.
    Anna Geller

    Anna Geller

    4 months ago
    I can totally understand you in that regard, managing servers is not something you should have to worry about if you are a data scientist - we are aware of the problem here and the solution we suggested in the shared discussion is to still use serverless - even still use Fargate - but exposed as a Kubernetes service. If EKS on Fargate is confusing, maybe you can check my old blog post about the topic here