How is everyone versioning the output based on which version of the flow/Task created it? I was thinking of recreating this approach in Prefect, where the
Task
object has a version and a hash that is a combination of the version of the current Task object and all Task before it. The output file will contain the final/flow's hash. Any ideas on how to approach this differently?
j
josh
08/26/2020, 11:38 AM
Hi @An Hoang optionally you could also template your Results using a callable (as shown here) and then that callable could be used to version the output 🙂
a
An Hoang
08/26/2020, 12:21 PM
thanks @josh. That provides a nice way for templating files. What about versioning at the Task level to determine which tasks should be ran and which to use cached result when some tasks change in a Flow? Is there a better way to do it than the way I described above?
An Hoang
08/26/2020, 12:30 PM
Oh I see Data Lineage as a coming soon feature of prefect cloud
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.