https://prefect.io
Join Slack
Hey all - Im trying to build somewhat of a self healing flow, but I’m having trouble thinking about ...
# ask-community
l
Hey all - Im trying to build somewhat of a self healing flow, but I’m having trouble thinking about the best design for the goal I’m trying to achieve (writing details in thread to prevent large messagE)
The flow will run hourly and fetch data for the previous hour + transform + deliver elsewhere. I’ve considered using some retry logic on certain tasks, however in a case where say one of my data sources is unreachable for a certain period of time between hours and retries don’t suffice. Then I thought about using the KV store to stash the last successful time interval, then having a parent flow that will kick off child flow runs for ever hour since the last successful 
scheduled_start_time
to ensure delivery of data remains chunked by 1 hour without doing excessive retries
k
This would be more straightforward if you could use the KV Store as a watermark and just process all the new data. Is the requirement to do it per hour chunk?
l
Unfortunately, yes - the consumer is expecting 1 hour chunks of data
That was my initial plan though
k
I think this is the right direction then
👍 1
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by