Hi i'm trying out prefect, I would need your guidance for my project.
My project is a series of "task" (would be flows in prefect language) lets say A ->B ->C ->D ->..., which are run sequentially
These tasks take as input huge files, they can't fit in RAM most of the time, so we load them in chunks. They also output big files, which are then passed to the other subflows downstream.
How should I manage the storage of these files (locally, uploading/downloading from buckets would be too long) leveraging prefect ? Taking into account that I may want to save intermediary results in case of a crash, or do stuff like this
A ->B ->C ->D ->...
| ^
v___________|
Where task D takes as input the output file of A and C.