Hiya :octavia-hi:
Quick question on triggering Flo...
# ask-communityj
Jovan Sakovic
06/14/2022, 11:12 PMHiya octavia hi
Quick question on triggering Flows, any thoughts or ideas are appreciated and welcome 🙌
We have a Jenkins job extracting data from Postgres postg and loading into BigQuery (yep, aware it could’ve been done with certain ETL tools, but it wasn’t an option)
We also have a few Airbyte syncs on the same VM that Prefect is running on too.
How would it make the most sense to trigger dbt jobs after the Jenkins and Airbyte syncs are finished? 🤔
Jovan Sakovic
06/14/2022, 11:13 PMObviously, if it was only the Airbyte syncs, we’d just use the AirbyteConnectionTask, and then set the dbt run job on successful runs. But how can we make sure that Jenkins job has finished too?
Jovan Sakovic
06/14/2022, 11:15 PMThought of 2 options so far, but they’re not the happiest solutions tbh 🙈
1. have a long running flow that queries BQ to check whether the data is as fresh as we want it to be, once it is, we trigger the dbt job (perhaps with Airbyte as a dependency to it)
2. add another command to the jenkins job that triggers the prefect flow (airbyte + dbt) through a
docker execprefect runk
Kevin Kho
06/15/2022, 1:27 AM
Hey @Jovan Sakovic, I would do number 2. You want the event to trigger the Prefect Flow, and have no schedule on the Flow. Especially here it sounds like you want to use Prefect 1 for the available tasks (Airbyte specifically). Long running flows are not really good in Prefect 1.0 compared to Prefect 2.0
3 Views