Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
z
Zack
12/01/2022, 3:44 PMIs there a comparable orchestration tool in AWS and what would be the advantages prefect has over it? Resources would be great.
p
Peyton Runyan
12/01/2022, 3:57 PMWhat features are you looking for? That's really going to dictate a lot of what makes one tool better than another.
z
Zack
12/01/2022, 4:01 PMEase of use, robust capability, ability to run aws services and pipelines like airbyte or fivetran if needed. What are the trade-offs in your mind?
p
Peyton Runyan
12/01/2022, 4:02 PMLol tons
AWS has step functions I think. It’s generally going to be less flexible and ease of use is pretty subjective.
I imagine both are sufficiently robust
In general prefect is going to be python native and platform independent. If you like working in python, I’ll imagine you’ll like working with prefect
❤️ 3
Prefect is pretty fast to get started with. Maybe take an hour and run through a tutorial and see if you like it? If you’ve got any questions feel free to ask here
I’m mobile at the moment but I’ll add some resources when I’m back from lunch
Actually - this article is a good place to start to get a feel for prefect’s approach: https://medium.com/the-prefect-blog/modular-data-stack-build-a-data-platform-with-prefect-dbt-and-snowflake-9e8ef6a56503
:gratitude-thank-you: 2
Step function docs: https://aws.amazon.com/step-functions/
👀 1
Prefect getting started tutorial - https://docs.prefect.io/getting-started/overview/
a
Anna Geller
12/01/2022, 4:30 PMplatform independentthis part explains the key diff - AWS is tied to AWS. Prefect isn't
🙌 1
m
Mike Grabbe
12/01/2022, 4:57 PMMy team is in the process of transitioning out of step functions and into Prefect. Step functions is actually a great choice when you need to link together several different AWS services, especially for event-driven workflows. However, we ran into several pain points with step functions that prompted us to make the switch:
• json definition files become too complex and difficult to manage
• built-in integrations only exist inside the aws ecosystem
• python needs to be wrapped inside lambda functions
• you need to build out and manage your own dependency graph. Prefect does this on its own.
:thank-you: 5
🙌 3
z
Zack
12/01/2022, 5:42 PMHow does prefect work with resource like fivetran or airbyte? Any resource for this? We intend to use dbt as well.
k
Kevin Grismore
12/01/2022, 5:46 PMPrefect has integrations with AWS, airbyte, fivetran, dbt and much more via their collections packages: https://docs.prefect.io/collections/catalog/
🙌 2
:thank-you: 2
z
Zack
12/01/2022, 6:24 PM@Kevin Grismore thanks for sharing this!
@Mike Grabbe Thank you for elaborating on your reasons for switching! I wonder did you ever consider MWAA and what were the reasons for going with step functions initially?
m
Mike Grabbe
12/01/2022, 6:34 PMwe tried out MWAA but quickly eliminated it as an option. AWS doesn't seem to prioritize development for that offering. For example, the latest Airflow version they support is v2.2.2, which Airflow released over a year ago. We couldn't get local development to work, and (non-aws) online resources/blog posts for MWAA were virtually non-existent.
🙌 3