Hi. I am a newbie and checking out the possibilities of prefect for our team. I was wondering whether there is a built in de-duplication of running a flow. What I mean is - if a particular flow (with given parameters) has been scheduled for a time T, and an event executes the flow with the same parameters within at T - dt (dt > 0) the scheduled flow is skipped (de-duplicated) (or vice versa for that matter)?
b
Bianca Hoch
12/09/2024, 4:57 PM
Hey Imi! In my mind, one way to go about this would be to add a task at the beginning of your flow script which checks to see if the inputs have been processed previously. If yes, have the flow complete and skip any downstream processes. If not, continue on with execution. I think using transactions to make your flow idempotent will help you out here.
i
imi
12/09/2024, 5:40 PM
Thanks. That I understand that by coding means it can be achieved. Will such an approach not report false statistics/history on the flow executed?
b
Bianca Hoch
12/09/2024, 6:41 PM
Will such an approach not report false statistics/history on the flow executed?
Would you mind sharing a bit more about what you mean by false statistics/history?
i
imi
12/09/2024, 10:32 PM
What I mean: How will a flow's status appear in the overview? It will have to get started and then cancelled (by transaction or other distributed lock means). Is there a way to make clear to the operator that a task has not failed, but was cancelled because of the recent execution of an identical flow.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.