Jovan Sakovic

    Jovan Sakovic

    3 months ago
    Hiya 😮ctavia-hi: Quick question on triggering Flows, any thoughts or ideas are appreciated and welcome 🙌 We have a Jenkins job extracting data from Postgres 😛ostg: and loading into BigQuery (yep, aware it could’ve been done with certain ETL tools, but it wasn’t an option) We also have a few Airbyte syncs on the same VM that Prefect is running on too. How would it make the most sense to trigger dbt jobs after the Jenkins and Airbyte syncs are finished? 🤔
    Obviously, if it was only the Airbyte syncs, we’d just use the AirbyteConnectionTask, and then set the dbt run job on successful runs. But how can we make sure that Jenkins job has finished too?
    Thought of 2 options so far, but they’re not the happiest solutions tbh 🙈 1. have a long running flow that queries BQ to check whether the data is as fresh as we want it to be, once it is, we trigger the dbt job (perhaps with Airbyte as a dependency to it) 2. add another command to the jenkins job that triggers the prefect flow (airbyte + dbt) through a
    docker exec
    +
    prefect run
    command
    Kevin Kho

    Kevin Kho

    3 months ago
    Hey @Jovan Sakovic, I would do number 2. You want the event to trigger the Prefect Flow, and have no schedule on the Flow. Especially here it sounds like you want to use Prefect 1 for the available tasks (Airbyte specifically). Long running flows are not really good in Prefect 1.0 compared to Prefect 2.0