Hey folks! I’ve been struggling a bit with understanding how Prefect connects with the AWS ecosystem. Namely ECR, ECS, and Fargate. I’d be willing to pay for a few hours of tutoring. Please DM me your rates :)
02/02/2021, 1:57 AM
I'm still on that journey, but I had a recent discussion that clarified some things for me, so I'll tag you there in case it helps.
02/02/2021, 2:10 AM
Hey @Jimmy Le,
Happy to oblige here.
Prefect empowers you to orchestrate your workflows using Python in many ways, but particularly in regards to third-party tooling and your chosen Cloud Platform, both of which can vary greatly. Prefect’s Hybrid Model ensures your code and data never leave your infrastructure to maintain the sanctity of your data, leaving you in charge of your code/data specifics like execution and code storage and that’s where AWS comes into play (or any Cloud Platform for that matter).
So typically, users in the AWS space will choose ECR for containerized code storage, to be retrieved by an ECS/Fargate Agent, to be executed on the cluster that the Agent lives on (outside of Prefect entirely), all the while relaying metadata metrics back to your Cloud UI. As a sidenote, if you need help with your infrastructure, we do have partners we work with so I can have someone reach out! (I work @ Prefect btw )
Hope that helps!
02/02/2021, 2:26 AM
I’m gonna give it one more go in the morning! Thank you @Amanda Wee and @Kyle Moon-Wright 😬
02/02/2021, 5:29 PM
Hey Jimmy, my name is Kyle DeMerritt (Kyle #2 in this case 🙂 ) and I can certainly help you with support options. Depending on your specific needs we have multiple options. Namely, our partners at Slate are a fantastic resource as you look to optimize your team's resources this year. Through our partnership, they are well equipped to help companies prepare, deploy and scale Prefect Cloud. This can include both infrastructure and Prefect Cloud implementation. If you provide your email address, I can certainly provide more information and make an introduction.
Please also feel free to email me directly - firstname.lastname@example.org
02/06/2021, 2:13 PM
Got it to work…! Now to try and reproduce what I’ve done 😅