Hi prefect friends, I haven’t checked in in a bit. Do we have
Result
equivalent in prefect 2.0? 🙂
✅ 1
Example use cases:
• Cloud store
• mlflow metrics
k
Khuyen Tran
09/20/2022, 3:16 PM
Do you want to add checkpoint to your task?
e
Evan Curtin
09/20/2022, 4:07 PM
I want to pass data from one task to the next using data storage outside of prefect
either because 1: data is far too large to handle in prefect or 2: I want it persisted outside anwyay
yeah and I’d like to use the Result type to allow custom checkpointing logic (i.e. things like checking for spark’s _SUCCESS file at a remote storage location)
b
Bianca Hoch
09/20/2022, 6:08 PM
Hey Evan, just trying to brainstorm potential solutions for what you're trying to accomplish here. Have you considered using Blocks to utilize storage external to Prefect? We have one which can be used for GCP.
I’ve been quite confused in 2.0 as to how I’m supposed to be passing data around between tasks. Am I expected to pass around URI’s and have each task do I/O?
b
Bianca Hoch
09/20/2022, 9:38 PM
Thank you for clarifying! We will have an equivalent for 1.0 Results in 2.0 in the near future. This is a high priority feature and will make a debut this quarter.