Thread
#prefect-community
    Matt Clapp

    Matt Clapp

    10 months ago
    Hi everyone, first post here and just coming up to speed on Prefect. So far it's a joy to use! My question is about running an agent on AWS, hopefully something like ECS. Specifically I'd like to have a durable setup/flow that stays on AWS, and totally cut my local computer out of the loop for everything. I see a lot of good tutorials that involve initiating a flow with a local computer that runs on AWS using ECSRun. (like https://towardsdatascience.com/serverless-data-pipelines-made-easy-with-prefect-and-aws-ecs-fargate-7e25bacb450c and https://www.lejimmy.com/distributed-data-pipelines-with-aws-ecs-fargate-and-prefect-cloud/) Tantalizingly, the second post by Jimmy Le has one mention that sounds close to my use case: "Ultimately we'll probably want to run many ECSAgents in a small Elastic Compute Cloud (EC2) with
    supervisord
    monitoring their health." What I'd like and what my question is: Is there documentation on just setting up a Docker image and having that be self-contained container running a flow on AWS, without needing to kick it off from a local computer? I'd be happy to use Prefect Cloud for UI. Is this a valid use case? Is there some reason there's not much documentation for it? (Or do I just not understand something basic?) thanks so much for any help.
    Kevin Kho

    Kevin Kho

    10 months ago
    Hey @Matt Clapp, welcome! You can run the ECSAgent as a service. The docs for that are here . This will make sure that if the agent dies, the service will be responsible for spinning up another instance. This is also containerized and can take care of deploying your flows
    Matt Clapp

    Matt Clapp

    10 months ago
    Ah thanks @Kevin Kho! I saw this part of the docs once but wasn't sure if it applied to me. This is helpful. I'll try it out, thanks again.
    @Kevin Kho one question, in this "ECSAgent as a service" scenario, how or when would I register my flow? Would I register it inside my custom Docker image? Or does it need to be registered after starting up the prefect agent? I'm still a little fuzzy on what actually happens when a flow is registered.
    I think I'm starting to understand. I just need to register a flow anywhere and as long as it has the proper labels (a subset of the labels on a given agent) it will run on that agent. (Let me know if I'm getting it wrong ๐Ÿ™‚ )
    Ah now I see this article that seems very appropriate. It is very detailed for the service flow: https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df I was thrown because the title is mainly talking about using Fargate Spot but really it is a good general article about doing what I want to do.
    Anna Geller

    Anna Geller

    10 months ago
    Let me know if Iโ€™m getting it wrong
    @Matt Clapp thatโ€™s correct. The only caveat is that ideally you should register your flow from an environment that has the same Prefect version as the one used in your execution environment to avoid any version mismatch.
    Matt Clapp

    Matt Clapp

    10 months ago
    btw @Anna Geller your article is awesome, very complete, very well explained. Thanks so much for writing it.