Sana Shaikh
07/28/2022, 5:23 PMNate
07/28/2022, 6:11 PMAirbyteConnectionTask
to trigger Airbyte syncs in a Prefect flow, as shown here or like below
from prefect.tasks.airbyte.airbyte import AirbyteConnectionTask
sync_airbyte_connection = AirbyteConnectionTask(
max_retries=3, retry_delay=timedelta(seconds=10)
)
# Sync Airbyte GitHub data (for example)
airbyte_github_sync = sync_airbyte_connection(
airbyte_server_host="localhost",
airbyte_server_port=8000,
airbyte_api_version="v1",
connection_id=airbyte_github_connection_id,
)
with Flow('My Airbyte Flow') as flow:
airbyte_github_sync() #this is a task
Lucien Fregosi
07/29/2022, 7:59 AMAirbyteConnectionTask
with prefect 2.0 ?Nate
07/29/2022, 3:13 PMAirbyteConnectionTask
directly since it is implemented as a prefect 1.0 task
however, we've already implemented the prefect-airbyte
collection (collections being the 2.0 replacement for the 1.0 task library) which has the trigger_sync
task (which does the same thing as AirbyteConnectionTask
)
here's the docs on usage!Lucien Fregosi
07/29/2022, 3:23 PM