Sven Teresniak

07/20/2020, 9:22 AM
Hi. I have a distributed Dask setup (DaskExecutor). I learned that I need to persist flows (and results) to be able to survice prefect cluster restarts. I added an NFS volume to share the exact same data between Agent and Dask Worker (these two only!) and added a
storage to the flow, pointing to a path in the NFS. Now the questions: 1. is it a good idea (or necessary?) to have the flows and results accessible in the same mountpoint (local filesystem location, e.g. NFS-persisted
) for Agent and Dask worker? Or is it sufficient to persist the flow/result storage (make it durable between restarts), and its not needed for the agent to access the pickeld flow results? 2. What's the pro and con of
for a
storage? When do I want to set this? Also, I do not understand the
parameter for local storage? I set
to a path on my NFS and now I see the pickled flows and results. What is path?