https://prefect.io logo
Join Slack
Powered by
Hi everyone, first post here and just coming up to...
# ask-community
m
Hi everyone, first post here and just coming up to speed on Prefect. So far it's a joy to use! My question is about running an agent on AWS, hopefully something like ECS. Specifically I'd like to have a durable setup/flow that stays on AWS, and totally cut my local computer out of the loop for everything. I see a lot of good tutorials that involve initiating a flow with a local computer that runs on AWS using ECSRun. (like https://towardsdatascience.com/serverless-data-pipelines-made-easy-with-prefect-and-aws-ecs-fargate-7e25bacb450c and https://www.lejimmy.com/distributed-data-pipelines-with-aws-ecs-fargate-and-prefect-cloud/) Tantalizingly, the second post by Jimmy Le has one mention that sounds close to my use case: "Ultimately we'll probably want to run many ECSAgents in a small Elastic Compute Cloud (EC2) with 
supervisord
monitoring their health." What I'd like and what my question is: Is there documentation on just setting up a Docker image and having that be self-contained container running a flow on AWS, without needing to kick it off from a local computer? I'd be happy to use Prefect Cloud for UI. Is this a valid use case? Is there some reason there's not much documentation for it? (Or do I just not understand something basic?) thanks so much for any help.
k
Hey @Matt Clapp, welcome! You can run the ECSAgent as a service. The docs for that are here . This will make sure that if the agent dies, the service will be responsible for spinning up another instance. This is also containerized and can take care of deploying your flows
upvote 1
m
Ah thanks @Kevin Kho! I saw this part of the docs once but wasn't sure if it applied to me. This is helpful. I'll try it out, thanks again.
@Kevin Kho one question, in this "ECSAgent as a service" scenario, how or when would I register my flow? Would I register it inside my custom Docker image? Or does it need to be registered after starting up the prefect agent? I'm still a little fuzzy on what actually happens when a flow is registered.
I think I'm starting to understand. I just need to register a flow anywhere and as long as it has the proper labels (a subset of the labels on a given agent) it will run on that agent. (Let me know if I'm getting it wrong 🙂 )
Ah now I see this article that seems very appropriate. It is very detailed for the service flow: https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df I was thrown because the title is mainly talking about using Fargate Spot but really it is a good general article about doing what I want to do.
🙌 1
a
Let me know if I’m getting it wrong
@Matt Clapp that’s correct. The only caveat is that ideally you should register your flow from an environment that has the same Prefect version as the one used in your execution environment to avoid any version mismatch.
👍 1
upvote 1
m
btw @Anna Geller your article is awesome, very complete, very well explained. Thanks so much for writing it.
🙌 1
7 Views
Open in Slack
PreviousNext
Powered by