Thread
#prefect-community
    Leon Kozlowski

    Leon Kozlowski

    9 months ago
    Hey all - Im trying to build somewhat of a self healing flow, but I’m having trouble thinking about the best design for the goal I’m trying to achieve (writing details in thread to prevent large messagE)
    The flow will run hourly and fetch data for the previous hour + transform + deliver elsewhere. I’ve considered using some retry logic on certain tasks, however in a case where say one of my data sources is unreachable for a certain period of time between hours and retries don’t suffice. Then I thought about using the KV store to stash the last successful time interval, then having a parent flow that will kick off child flow runs for ever hour since the last successful
    scheduled_start_time
    to ensure delivery of data remains chunked by 1 hour without doing excessive retries
    Kevin Kho

    Kevin Kho

    9 months ago
    This would be more straightforward if you could use the KV Store as a watermark and just process all the new data. Is the requirement to do it per hour chunk?
    Leon Kozlowski

    Leon Kozlowski

    9 months ago
    Unfortunately, yes - the consumer is expecting 1 hour chunks of data
    That was my initial plan though
    Kevin Kho

    Kevin Kho

    9 months ago
    I think this is the right direction then