https://prefect.io logo
Title
s

Sana Shaikh

07/28/2022, 5:23 PM
Hello folks. Can you answer or provide any example related to my question!. 1. How to run the cloud pipeline on prefect-server? 2. How to run the airbyte task with prefect pipeline Would highly appreciate help from this community šŸŽ‰
āœ… 1
n

Nate

07/28/2022, 6:11 PM
Hi @Sana Shaikh ! ā€¢ I'm not sure what you mean by "*cloud pipeline on prefect-server",* but here is some information on running prefect server (which I should note will not be as well-maintained compared to Prefect Cloud 1.0 and 2.0 in the future) ā€¢ in Prefect 1.0, you can use the
AirbyteConnectionTask
to trigger Airbyte syncs in a Prefect flow, as shown here or like below
from prefect.tasks.airbyte.airbyte import AirbyteConnectionTask

sync_airbyte_connection = AirbyteConnectionTask(
    max_retries=3, retry_delay=timedelta(seconds=10)
)

# Sync Airbyte GitHub data (for example)
airbyte_github_sync = sync_airbyte_connection(
    airbyte_server_host="localhost",
    airbyte_server_port=8000,
    airbyte_api_version="v1",
    connection_id=airbyte_github_connection_id,
)

with Flow('My Airbyte Flow') as flow:
   airbyte_github_sync() #this is a task
āœ… 1
l

Lucien Fregosi

07/29/2022, 7:59 AM
hi @Nate And it it possible to use
AirbyteConnectionTask
with prefect 2.0 ?
n

Nate

07/29/2022, 3:13 PM
hi @Lucien Fregosi, you can't use
AirbyteConnectionTask
directly since it is implemented as a prefect 1.0 task however, we've already implemented the
prefect-airbyte
collection (collections being the 2.0 replacement for the 1.0 task library) which has the
trigger_sync
task (which does the same thing as
AirbyteConnectionTask
) here's the docs on usage!
l

Lucien Fregosi

07/29/2022, 3:23 PM
Awesome thanks @Nate