Any recommendations on the best way to handle statefulness between flow runs and tasks.
I’m looking to make a flow which executes many steps fault tolerant in case a particular task and/or step fails.
Tools such as temporal handle this well, but requires a lot of heavy lifting to implement. I’m wondering if there is a best practice approach available within Prefect.
Use Case:
I have a pipeline which contains many tasks that updates a Database record as it progresses through the flow.
A failure during this workflow would leave a record in a potentially broken state where a status has been updated, but has not progressed to a subsequent step.
A re-run of this flow without proper statefulness could potentially render this record as broken forever without manual intervention to reset the record.