Is there a comparable orchestration tool in AWS an...
# data-tricks-and-tipsz
Zack
12/01/2022, 3:44 PMIs there a comparable orchestration tool in AWS and what would be the advantages prefect has over it? Resources would be great.
p
Peyton Runyan
12/01/2022, 3:57 PMWhat features are you looking for? That's really going to dictate a lot of what makes one tool better than another.
z
Zack
12/01/2022, 4:01 PMEase of use, robust capability, ability to run aws services and pipelines like airbyte or fivetran if needed. What are the trade-offs in your mind?
p
Peyton Runyan
12/01/2022, 4:02 PMLol tons
Peyton Runyan
12/01/2022, 4:05 PMAWS has step functions I think. It’s generally going to be less flexible and ease of use is pretty subjective.
Peyton Runyan
12/01/2022, 4:07 PMI imagine both are sufficiently robust
Peyton Runyan
12/01/2022, 4:08 PMIn general prefect is going to be python native and platform independent. If you like working in python, I’ll imagine you’ll like working with prefect
❤️ 3
Peyton Runyan
12/01/2022, 4:11 PMPrefect is pretty fast to get started with. Maybe take an hour and run through a tutorial and see if you like it? If you’ve got any questions feel free to ask here
Peyton Runyan
12/01/2022, 4:16 PMI’m mobile at the moment but I’ll add some resources when I’m back from lunch
Peyton Runyan
12/01/2022, 4:17 PMActually - this article is a good place to start to get a feel for prefect’s approach: https://medium.com/the-prefect-blog/modular-data-stack-build-a-data-platform-with-prefect-dbt-and-snowflake-9e8ef6a56503
gratitude thank you 2
Peyton Runyan
12/01/2022, 4:18 PMStep function docs: https://aws.amazon.com/step-functions/
👀 1
Peyton Runyan
12/01/2022, 4:19 PMPrefect getting started tutorial - https://docs.prefect.io/getting-started/overview/
a
Anna Geller
12/01/2022, 4:30 PM
platform independentthis part explains the key diff - AWS is tied to AWS. Prefect isn't
🙌 1
m
Mike Grabbe
12/01/2022, 4:57 PMMy team is in the process of transitioning out of step functions and into Prefect. Step functions is actually a great choice when you need to link together several different AWS services, especially for event-driven workflows. However, we ran into several pain points with step functions that prompted us to make the switch:
• json definition files become too complex and difficult to manage
• built-in integrations only exist inside the aws ecosystem
• python needs to be wrapped inside lambda functions
• you need to build out and manage your own dependency graph. Prefect does this on its own.
🙏 5
🙌 3
z
Zack
12/01/2022, 5:42 PMHow does prefect work with resource like fivetran or airbyte? Any resource for this? We intend to use dbt as well.
k
Kevin Grismore
12/01/2022, 5:46 PMPrefect has integrations with AWS, airbyte, fivetran, dbt and much more via their collections packages: https://docs.prefect.io/collections/catalog/
🙌 2
🙏 2
z
Zack
12/01/2022, 6:24 PM@Kevin Grismore thanks for sharing this!
Zack
12/01/2022, 6:26 PM@Mike Grabbe Thank you for elaborating on your reasons for switching! I wonder did you ever consider MWAA and what were the reasons for going with step functions initially?
m
Mike Grabbe
12/01/2022, 6:34 PMwe tried out MWAA but quickly eliminated it as an option. AWS doesn't seem to prioritize development for that offering. For example, the latest Airflow version they support is v2.2.2, which Airflow released over a year ago. We couldn't get local development to work, and (non-aws) online resources/blog posts for MWAA were virtually non-existent.
🙌 3
5 Views