Ihor Ramskyi
09/15/2025, 3:03 PMMarvin
09/15/2025, 3:03 PMMarvin
09/15/2025, 3:10 PMCachePolicy(key_storage=...)
- Persisted results: controlled separately by your task/flow result_storage
(and whether persist_result
is enabled)
So if you changed key_storage
, new cache records will go to that new folder — but:
- Old cache records will remain in the old folder (Prefect doesn’t move them automatically)
- If you’re persisting results, those files will still go to the default result storage unless you change that too
How to change the cache record location
- Set key_storage
on the cache policy you pass to your tasks. You can use a path or a filesystem:
from prefect import task
from prefect.cache_policies import CachePolicy
from prefect.filesystems import LocalFileSystem
# Option A: plain path
my_cache = CachePolicy(key_storage="/mnt/prefect/cache")
# Option B: LocalFileSystem
# my_cache = CachePolicy(key_storage=LocalFileSystem(path="/mnt/prefect/cache"))
@task(cache_policy=my_cache)
def compute(...):
...
If you want all cached tasks to write to the same place, make sure every task that uses caching gets this same CachePolicy
. Otherwise some tasks will still write to the default location.
If you also want to move persisted results
- Set result storage at the flow level so it applies to tasks, and enable persistence if needed:
from prefect import flow
@flow(result_storage="/mnt/prefect/results") # set where results are stored
def my_flow():
...
Housekeeping
- It’s safe to delete old cache records if you don’t need them anymore (it will just force recomputation the next time those keys are requested).
- If you’re seeing files in two locations, it’s usually:
- legacy cache records in the old folder, plus new cache records in your new key_storage
- and/or cache records in one location and persisted results in another
Docs for reference
- Caching in Prefect 3
- Results and result storage
Can you share the two paths you’re seeing and a snippet showing how you configured CachePolicy
? That’ll let me confirm whether it’s cache vs result files or a mix of default and custom policies.Ihor Ramskyi
09/16/2025, 11:56 AMMarvin
09/16/2025, 11:56 AMMarvin
09/16/2025, 12:01 PMresult_storage
can be a filesystem block, a saved block reference (UUID), or a pathlib.Path. If passing a path is erroring, the most reliable fix is to pass a filesystem object explicitly.
Try one of these patterns:
- Local directory via a filesystem object
from prefect import flow
from prefect.filesystems import LocalFileSystem
@flow(result_storage=LocalFileSystem(basepath="/tmp/prefect-results"), persist_result=True)
def my_flow():
return 42
- Local directory via Path
from pathlib import Path
from prefect import flow
@flow(result_storage=Path("/tmp/prefect-results"), persist_result=True)
def my_flow():
return 42
- Saved block you can reuse across runs/environments
from prefect.filesystems import LocalFileSystem
# one-time setup
LocalFileSystem(basepath="/tmp/prefect-results").save("results-local", overwrite=True)
# later
from prefect import flow
@flow(result_storage=LocalFileSystem.load("results-local"), persist_result=True)
def my_flow():
return 42
Notes:
- Set persist_result=True
on the flow (and/or tasks) if you actually want values written to storage.
- Ensure the directory exists and your worker has write access if you use a local path.
- For remote storage, pass the appropriate filesystem block (e.g., S3/GCS/Azure from the integrations) or a saved block reference.
If this still raises an error, can you share:
- Your Prefect version (prefect version
)
- The exact code snippet where you set result_storage
- The full error traceback
That’ll help me pinpoint whether it’s a parsing issue (e.g., Pydantic union) or an environment/permissions problem.