https://prefect.io
Join Slack
Menu
Home
Archive
Is there a comparable orchestration tool in AWS and what would be the advantages prefect has over it...
# data-tricks-and-tips
z
Is there a comparable orchestration tool in AWS and what would be the advantages prefect has over it? Resources would be great.
p
What features are you looking for? That's really going to dictate a lot of what makes one tool better than another.
z
Ease of use, robust capability, ability to run aws services and pipelines like airbyte or fivetran if needed. What are the trade-offs in your mind?
p
Lol tons
AWS has step functions I think. It’s generally going to be less flexible and ease of use is pretty subjective.
I imagine both are sufficiently robust
In general prefect is going to be python native and platform independent. If you like working in python, I’ll imagine you’ll like working with prefect
❤️ 3
Prefect is pretty fast to get started with. Maybe take an hour and run through a tutorial and see if you like it? If you’ve got any questions feel free to ask here
I’m mobile at the moment but I’ll add some resources when I’m back from lunch
Actually - this article is a good place to start to get a feel for prefect’s approach: https://medium.com/the-prefect-blog/modular-data-stack-build-a-data-platform-with-prefect-dbt-and-snowflake-9e8ef6a56503
gratitude thank you 2
👀 1
Prefect getting started tutorial - https://docs.prefect.io/getting-started/overview/
a
platform independent
this part explains the key diff - AWS is tied to AWS. Prefect isn't
🙌 1
m
My team is in the process of transitioning out of step functions and into Prefect. Step functions is actually a great choice when you need to link together several different AWS services, especially for event-driven workflows. However, we ran into several pain points with step functions that prompted us to make the switch: • json definition files become too complex and difficult to manage • built-in integrations only exist inside the aws ecosystem • python needs to be wrapped inside lambda functions • you need to build out and manage your own dependency graph. Prefect does this on its own.
🙏 5
🙌 3
z
How does prefect work with resource like fivetran or airbyte? Any resource for this? We intend to use dbt as well.
k
Prefect has integrations with AWS, airbyte, fivetran, dbt and much more via their collections packages: https://docs.prefect.io/collections/catalog/
🙌 2
🙏 2
z
@Kevin Grismore thanks for sharing this!
@Mike Grabbe Thank you for elaborating on your reasons for switching! I wonder did you ever consider MWAA and what were the reasons for going with step functions initially?
m
we tried out MWAA but quickly eliminated it as an option. AWS doesn't seem to prioritize development for that offering. For example, the latest Airflow version they support is v2.2.2, which Airflow released over a year ago. We couldn't get local development to work, and (non-aws) online resources/blog posts for MWAA were virtually non-existent.
🙌 3
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by