Hi everyone - I'm trying to use the persist/cache features of a task runner to store results between flow executions - is this possible, or is the data only persisted within a specific flow/subflow execution
j
Janet Carson
02/12/2024, 6:48 PM
Prefect's persist feature is for prefect's internal use -- if you persist a result and then rerun the same flow, the server will see the same task with the same arguments and get results out of the cache. If you want to store something for some other purpose besides failure recovery, I would personally suggest storing it in your own defined storage area with your own defined format and naming convention. If you insist on using the persist feature, make sure that it is a small amount of state as Prefect insists on Base64 JSON encoding for persisting results which will cause something like a large binary data blob to bloat up.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.