Is there a way to trigger a Prefect task after a dbt job finishes that's scheduled by dbt cloud? In the future we might manage dbt jobs using Prefect. For now we're looking for a solution / workaround where dbt jobs are scheduled by dbt. The obvious workaround is scheduling Prefect tasks for a time when we know dbt will be finished. Are there any alternatives?

Hey @Mathijs Miermans, not super familiar but does dbt Cloud offer webhooks? Because you would just need to hit the

create_flow_run

API endpoint.

Thanks for the pointers! Another workaround would be to create a polling Prefect task, that waits until dbt is finished: https://github.com/PrefectHQ/prefect/discussions/3944 Is it still accurate that there are no major pitfalls with this approach, and that the best way to do this is creating a simple loop in the task?

while True: 
    time.sleep(10)
    if dbt_job.is_finished():
        break
...

No. You can poll in a task for sure. We have tasks in the task library that do the same.

We actually have a DbtCloudRunJob task and there is a blog post on the roadmap explaining how to use it

Awesome! Could you point me to an example?

What Anna posted will poll for you

Awesome. That seems exactly what we need. Thanks again for the quick and helpful support! Best support ever! 🙇

@Mathijs Miermans here is the walkthrough - just published https://www.prefect.io/blog/orchestrating-elt-with-prefect-dbt-cloud-and-snowflake-part-3/

Very timely. I'll make a note to reconsider whether we can trigger dbt cloud jobs from Prefect. It seems very simple.