Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hey all - Im trying to build somewhat of a self healing flow, but I’m having trouble thinking about the best design for the goal I’m trying to achieve (writing details in thread to prevent large messagE)

The flow will run hourly and fetch data for the previous hour + transform + deliver elsewhere.

I’ve considered using some retry logic on certain tasks, however in a case where say one of my data sources is unreachable for a certain period of time between hours and retries don’t suffice.

Then I thought about using the KV store to stash the last successful time interval, then having a parent flow that will kick off child flow runs for ever hour since the last successful `scheduled_start_time` to ensure delivery of data remains chunked by 1 hour without doing excessive retries

This would be more straightforward if you could use the KV Store as a watermark and just process all the new data. Is the requirement to do it per hour chunk?

Unfortunately, yes - the consumer is expecting 1 hour chunks of data