Is there a comparable orchestration tool in AWS and what wou

Question

Is there a comparable orchestration tool in AWS and what would be the advantages prefect has over it Resources would be great

Peyton Runyan · Answer

What features are you looking for That s really going to dictate a lot of what makes one tool better than another

Zack · Answer

Ease of use robust capability ability to run aws services and pipelines like airbyte or fivetran if needed What are the trade offs in your mind

Peyton Runyan · Answer

Lol tons

Peyton Runyan · Answer

AWS has step functions I think It s generally going to be less flexible and ease of use is pretty subjective

Peyton Runyan · Answer

I imagine both are sufficiently robust

Peyton Runyan · Answer

In general prefect is going to be python native and platform independent If you like working in python I ll imagine you ll like working with prefect

Peyton Runyan · Answer

Prefect is pretty fast to get started with Maybe take an hour and run through a tutorial and see if you like it If you ve got any questions feel free to ask here

Peyton Runyan · Answer

I m mobile at the moment but I ll add some resources when I m back from lunch

Peyton Runyan · Answer

Actually this article is a good place to start to get a feel for prefect s approach <https medium com the prefect blog modular data stack build a data platform with prefect dbt and snowflake 9e8ef6a56503|https medium com the prefect blog modular data stack build a data platform with prefect dbt and snowflake 9e8ef6a56503>

Peyton Runyan · Answer

Step function docs <https aws amazon com step functions |https aws amazon com step functions >

Peyton Runyan · Answer

Prefect getting started tutorial <https docs prefect io getting started overview |https docs prefect io getting started overview >

Anna Geller · Answer

gt platform independent this part explains the key diff AWS is tied to AWS Prefect isn t

Mike Grabbe · Answer

My team is in the process of transitioning out of step functions and into Prefect Step functions is actually a great choice when you need to link together several different AWS services especially for event driven workflows However we ran into several pain points with step functions that prompted us to make the switch json definition files become too complex and difficult to manage built in integrations only exist inside the aws ecosystem python needs to be wrapped inside lambda functions you need to build out and manage your own dependency graph Prefect does this on its own

Zack · Answer

How does prefect work with resource like fivetran or airbyte Any resource for this We intend to use dbt as well

Kevin Grismore · Answer

Prefect has integrations with AWS airbyte fivetran dbt and much more via their collections packages <https docs prefect io collections catalog >

Zack · Answer

< Kevin Grismore> thanks for sharing this

Zack · Answer

< Mike Grabbe> Thank you for elaborating on your reasons for switching I wonder did you ever consider MWAA and what were the reasons for going with step functions initially

Mike Grabbe · Answer

we tried out MWAA but quickly eliminated it as an option AWS doesn t seem to prioritize development for that offering For example the latest Airflow version they support is v2 2 2 which Airflow released over a year ago We couldn t get local development to work and non aws online resources blog posts for MWAA were virtually non existent