Hey all - I have a question about how to do something in Prefect. I want to kick off a particular flow once 1) we read a request for some work from a queue in the cloud AND 2) a certain set of flow dependencies have succeeded. Currently, the way we achieve this is by polling for success of the dependent flows in the flow triggered after reading the request from the queue. This isn't great, obviously, since it wastes workers and doesn't proceed as fast as it should once dependent flows complete. Any recommendations about how to do this more elegantly in Prefect?

Dependencies are by default implicit in Prefect. job_info = read_queue() dep1_result = dependency_one() dep2_result = dependency_two() submit_new_flow(job_info, dep1_result, dep2_result) It becomes the orchestrator's job to decide when all the dependencies are available for the call to submit_new_flow Can you say a bit about why that pattern isn't working for you?

Thanks for answering! I want to be able to wait around for the dependent flow to finish and kick off a new flow once the dependent flow succeeds. In the code snippet you shared, there’s nothing that waits for the dependency and then proceeds. One way to wait is just to poll the Prefect API periodically, but I’m wondering if there’s a cleaner way. Does that make sense? I might not be explaining well.

It doesn't quite make sense to me... dependency_one returns a result, so the orchestrator has to wait for that dependency to complete and return that result to pass as an argument to submit_new_flow. The way I drew it above, dependency_two doesn't depend on the result from dependency_one so that could be orchestrated to run concurrently.

Maybe this is the disconnect: the flow that reads the requests is not responsible for kicking off the dependent flow. We might have N of these dependent flows kicked off by separate processes. Then we might have M requests, each of which needs to confirm that some of those N flows succeeded. Does that clarify things?

Look at creating an automation on the change of flow status? https://docs.prefect.io/v3/how-to-guides/automations/creating-automations

Thanks! Yeah, automations might be a good solution. Have to look into that more. I think I need one that is triggered only when multiple flows have a state of Completed. Also not sure how to handle the situation where all dependent flows are completed before the automation is created.

Automations are generally on the deployment/flow itself, and then you'd have to figure out which runs of that flow are in which "watch groups" to decide whether to trigger the next thing.