Hey folks, is there a way to make a Prefect task that triggers an external managed service wait for its completion? Say that a task triggers a pipeline on Rivery. How can I wait for the pipeline to finish running, before completing the task? Do I have to poll Rivery APIs ?

Hey frederico, thanks for the question. You could indeed poll the rivery API and this is likely what many people do today. We do have some early stage features that could help with this in prefect cloud via events and automations. You’ll need to a) be on prefect cloud and b) need some way to have the rivery system make an api call to prefect when its job is done. Can you have whatever is happening in rivery run a line of python code that emits an event to prefect? Alternatively, can rivery call webhooks? We will, in a few weeks, introduce the ability to have a prefect webhook url that external services can hit.

Alternatively, can rivery call webhooks?

Possibly yes. Rivery tbh was just an example for any managed EL service. Sadly, having tested about a dozen of them, not all implement webhooks or other solutions to send an event when a job execution finishes. Some do, some don't, which is quite annoying if you ask me because it severely limits the ways someone can interact with those services. Example: Hevodata doesn't have neither webhooks, nor api calls.

yeah no worries, feel free to keep in touch as you continue to solve this problem.