Hi prefect friends, I haven’t checked in in a bit. Do we have
equivalent in prefect 2.0? 🙂
Example use cases:
• Cloud store
• mlflow metrics
09/20/2022, 3:16 PM
Do you want to add checkpoint to your task?
09/20/2022, 4:07 PM
I want to pass data from one task to the next using data storage outside of prefect
either because 1: data is far too large to handle in prefect or 2: I want it persisted outside anwyay
yeah and I’d like to use the Result type to allow custom checkpointing logic (i.e. things like checking for spark’s _SUCCESS file at a remote storage location)
09/20/2022, 6:08 PM
Hey Evan, just trying to brainstorm potential solutions for what you're trying to accomplish here. Have you considered using Blocks to utilize storage external to Prefect? We have one which can be used for GCP.