Hi all, I’m new here and decided to learn Prefect but are anyone using it with Airbyte? I just want to know if there is a common best practice for that.

Hi @takan, I have not used Airbyte but I took a look at it for a bit. I think Airbyte gives a lot of off the shelf connections for data replication. I think the transform portion though is primarily limited to SQL. There is a couple of ways Prefect can integrate with Airbyte 1. Use the Airbyte connectors to construct rapid Extract-Load pipelines. Use Prefect as a scheduler to run these jobs by hitting the Airbyte API. Prefect as a workflow orchestration tool provides more state handling, dependency management, failure notification. 2. Use Prefect for the transformation steps in Airbyte. Because Prefect is written in Python, you have a lot more capabilities in terms of transformations. It is also a lot easier to recycle Python functions as modules. 3. Use Prefect to chain together operations that are not limited to data replication. Think about having an ETL job and then running a machine learning model training job, and then deploying a model. In the process you can output logs to DataDog or persist artifacts in MLFlow. Prefect is a lot more flexible because it’s written in Python so anything can be a task in Prefect as long as it can be done in Python. You can also trigger AWS jobs, or cross-cloud jobs for that matter.

Thanks for the reply @Kevin Kho! Okay I got each point 🙂 I’ll play around with Prefect to combine both awesome tools in our data stuck!