Is Prefect angling for an in-road to join Airflow as a managed solution by AWS? I'm asking because I've recently joined a new company, and the current plan is to use AWS managed Airflow to orchestrate dbt. I'm still learning, but my assessment is that it doesn't make sense to propose Prefect as an alternative right now because what the team is doing with Airflow is so simple. But in the future, there may be more, in which case I expect to run into the lack of AWS managed Prefect as an obstacle.

First of all, congrats on your new job! 🎉 One-click deployment of agents is on our radar for 2.0 - whether from the official AWS marketplace or in some other way, this is still WIP. As much as I would love to recommend you to start right away with Prefect 2.0, if MWAA perfectly fits your team, I wouldn't argue. But I think the main challenge in choosing a tool is choosing it not only for the current use case but also for not-yet-known future use cases, as well as matching your team (i.e. product shouldn't only solve a problem but should be also ergonomic and suit the user) I definitely understand your situation here fwiw dbt shell task from 1.0 is currently being migrated to 2.0 as part of this PR so your use case will soon be easily doable with Prefect 2.0, both OSS and Cloud

Thanks! I beat @Kevin Kho to starting a new adventure first! 😂 My team's airflow+dbt solution is still under development, but what I found interesting is that they plan to skip writing Python, and instead use Airflow DAG factory to allow us to write YAML configs to represent the DAGs with dbt tasks. To me, this negates any advantage that Prefect would provide. However, the data insights lead remarked that we might want to look into using less dbt and more airflow in the (unforeseen) future, which makes me think prefect may eventually be a good choice. My thinking now is that I may look into contributing a compiler to translate airflow dag factory config files to prefect 2.0. This way, assuming both dbt and snowflake task support are in prefect 2.0 by the time my chance presents itself, I'll be in a good position to make my case.

use Airflow DAG factory to allow us to write YAML configs to represent the DAGs with dbt tasks. To me, this negates any advantage that Prefect would provide.

this is also possible (and much easier!) in Prefect 2.0 - here is an example https://prefecthq.github.io/prefect-openmetadata/run_ingestion_flow/ But what I meant more is: what if you end up having one edge case where you use more than dbt and this YAML DAG factory is not sufficient? I understand the ease of use of config files, but if I had to choose would pick a tool that is equipped to handle more use cases in the future - you can add constraints only when constraints are needed, rather than choosing a tool that is constrained by default and doesn't allow to move past those constraints when some use case requires that -- this is how I would communicate it with your team - showing that you care about choosing the right tool that will work well long-term rather than one that will work just for now

You did lol. Congrats on the new job!