Andrew Schechtman-Rook12/23/2019, 8:13 PM
. I've tried a bunch of things to try to do this, including: • Turning on caching, and specifying
as the task/flow's
. In this case caching works, but it doesn't use the
to store/retrieve the task results. • Turning on
. This works to save the result to disk, but that appears to be a one way operation - AFAICT there aren't any hooks in prefect to pull the checkpointed data back in when restoring from cache. • Using custom `state_handler`s to essentially intercept the results and do the file I/O. I'll admit I've spent the least time on this approach, it seems like it might work but I don't have a good enough grasp on prefect to know either how to implement it or even if it's a good idea. Anyone had to implement something like this, or have any additional ideas?