Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Say I have a flow that processes data and publishes a manual computer vision annotation job to an annotation server - how might I schedule a flow to poll the annotation job to publish some information once the job has been completed?  It may take hours, days or maybe a week for this to finish, so polling every 4 hours or something may make sense.   Would this be an application for `pause_flow_run` using the `reschedule` argument?  Or is there another way I can create a `cron` like schedule for a parameterized flow to run until it successfully completes?

Alternatively, I could have a flow that tracks down the jobs/datasets that need to be published, but unfortunately that is a little bit harder that scheduling a polling operation

If i'm understanding right, would it work for your use case to kick off another flow run to publish the information once your original flow run completes? This way you wouldn't need to poll at all

theres a few ways to do this:

You can call `run_deployment()` inside of your flow run, maybe inside of the `on_completion` <https://docs.prefect.io/2.11.3/concepts/states/#state-change-hooks|hook>. here is a <https://github.com/PrefectHQ/prefect-recipes/blob/main/flows-advanced/parent-orchestrator/pokemon_weight.py|general example >for how to use run deployment.

Additionally if you're using Prefect Cloud, use can use <https://docs.prefect.io/2.11.3/concepts/automations/|automations>, where you can trigger a flow to kick off after success of another flow

That's correct, but I don't really know when the annotation job will be ready to pull down.  So something like `run_deployment` , maybe I could set retries to a large number and poll the status until it is completed that way?

Ah got it, so theres kind of a hidden middle step?
• flow1 completes
• annotation job (not a flow) happens in some indeterminate amount of time
• flow2 needs to run

Right, think of some middle blob of time that is on the scale of human work (days/week) and wrangling a web hook out of the annotation engine would be quite a bit of effort.