Hiya octavia hi Quick question on triggering Flows any thoug

Question

Hiya octavia hi Quick question on triggering Flows any thoughts or ideas are appreciated and welcome raised hands We have a Jenkins job extracting data from Postgres postg and loading into BigQuery yep aware it could ve been done with certain ETL tools but it wasn t an option We also have a few Airbyte syncs on the same VM that Prefect is running on too How would it make the most sense to trigger dbt jobs after the Jenkins and Airbyte syncs are finished thinking face

Jovan Sakovic · Answer

Obviously if it was only the Airbyte syncs we d just use the AirbyteConnectionTask and then set the dbt run job on successful runs But how can we make sure that Jenkins job has finished too

Jovan Sakovic · Answer

Thought of 2 options so far but they re not the happiest solutions tbh see no evil 1 have a long running flow that queries BQ to check whether the data is as fresh as we want it to be once it is we trigger the dbt job perhaps with Airbyte as a dependency to it 2 add another command to the jenkins job that triggers the prefect flow airbyte + dbt through a `docker exec` + `prefect run` command

Kevin Kho · Answer

Hey < Jovan Sakovic> I would do number 2 You want the event to trigger the Prefect Flow and have no schedule on the Flow Especially here it sounds like you want to use Prefect 1 for the available tasks Airbyte specifically Long running flows are not really good in Prefect 1 0 compared to Prefect 2 0