Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

I have a `pipelinewise` command as a shell task in a prefect flow. The command takes config files and a state file as an input parameter. The command executes and updates the state file at the end. What is the best way to retain the state file and also read the latest state file before executing the task?

Hi <@U01UJS353PZ>, we have a new feature that will help you with that called the KVStore.This <https://www.prefect.io/blog/introducing-the-prefect-kv-store|blog> is here. You can update the filename with the latest file.

Hmmm. I think KV store won't meet my requirement. Because pipelinewise integration will generate at least 6 files inside subfolders and these files will multiple with each integration. Also I'll need to preserve the folder path for the CLI to read the configuration.

In that case, I think you need to preserve the path with the KV store and then save/load that file somewhere like S3