Hey, is there a way for a flow to depend on anothe...
# prefect-community
j
Hey, is there a way for a flow to depend on another flow, with a different schedule. As in, wait execution until the first flow has completed (and vice versa)? I’ve seen running dependent flows topic, but that is constructing a higher-level flow. Where I want to have different schedules. Reason I need this: I sync several databases to BQ every 20 minutes, but on another schedule (once per day), I actually process the data in BQ. The data processing can’t run while sync are still running (due inconsistent data). So I have two crons, on different schedules, on which I want to create a dependency. Basically I want to hold a lock / semaphore, such that a flow waits until another flow releases it. Are there any suggestions to achieve this, preferably with core/server version? I’m not sure, but maybe what I’m looking for is described here in the cloud docs? And might it be implemented in this PR? Or do these tags behave differently?
1
j
Currently task concurrency limiting and flow concurrency limiting are available in Prefect Cloud for certain tiers. Either of the features in Cloud sound like they would solve your use case. The open PR on the server repo is a community driven implementation of flow-based concurrency that would also solve what you are after! I see that you commented on the issue and maybe we can push it over the line! 🙂
j
Cool, thanks for clearing that up 🙂 Lets hope the PR can be finished then.
a
@Joël Luijmes I faced the same problem, organizing order of flows on different schedules. And i came up with two tasks: One is waits for time to pass (uses schedule object), other waits for specified flow to have successful run in current UTC day. @josh Maybe i can contribute them to prefect task library? 🤔
j
Cool thanks for sharing! I’d say would be nice to have this in the task library 🙂
j
@Alexander yeah these would make great contributions!