<@ULVA73B9P> what is saved under /home/&lt;user&gt...
# ask-marvin
y
@Marvin what is saved under /home/<user>/.prefect/storage and how to do housekeeping for it ?
m
thought for 122.3 seconds
In Prefect 3.x,
~/.prefect/storage
is typically local result storage. That usually means Prefect writes serialized task/flow results there when result persistence is enabled and you're using the default local filesystem instead of remote storage. What you’ll find there - Files containing persisted task/flow return values - Cached results used for task caching / cache reuse - Metadata about how the result was serialized and whether it expires If you have not enabled result persistence or caching, this directory may be empty or absent. When Prefect uses it Prefect writes to
~/.prefect/storage
when: -
persist_result=True
is set on a flow or task - caching is enabled for a task -
PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
- no remote result storage block is configured, so Prefect falls back to local storage By default, the path is derived from Prefect home: -
PREFECT_HOME
defaults to
~/.prefect
- local storage defaults to
~/.prefect/storage
You can change it with
PREFECT_LOCAL_STORAGE_PATH
. Is it safe to delete? Yes, generally yes, but with caveats. Safe-ish: - deleting old files when no flows are actively running - deleting files you no longer need for cache reuse or result inspection Effects of deleting: - cached tasks may stop getting cache hits and re-run instead - retries or downstream logic that expects persisted local results may not find them - Prefect generally treats missing local results as “not available”, rather than catastrophically failing in all cases Not safe / not recommended: - deleting files during active runs - deleting files if you rely on persisted local results for retries/recovery - deleting them in ephemeral/distributed environments without understanding the impact Housekeeping recommendations Prefect does not currently do built-in garbage collection for this directory, so housekeeping is usually manual. Common options: 1. Check size
Copy code
bash
du -sh ~/.prefect/storage
2. Delete old files by age
Copy code
bash
find ~/.prefect/storage -type f -mtime +30 -delete
3. Delete everything if you do not need cached/persisted local results
Copy code
bash
rm -rf ~/.prefect/storage/*
Only do this when you’re confident nothing active depends on it. 4. Move storage somewhere else Set:
Copy code
bash
export PREFECT_LOCAL_STORAGE_PATH=/path/with/more/space
Or via Prefect config:
Copy code
bash
prefect config set PREFECT_LOCAL_STORAGE_PATH='/path/with/more/space'
5. Use remote result storage for better lifecycle management If these files are growing a lot, consider storing results in S3/GCS/Azure instead of local disk. Important caveat If you run flows in Docker/Kubernetes/ephemeral workers, local storage is often a poor fit for durable results because files disappear with the container/pod. In that case, remote result storage is usually better. Short version -
~/.prefect/storage
= local persisted results/cache artifacts - safe to clean up when no active runs depend on it - deleting files mainly causes cache misses / loss of persisted local results - housekeeping is manual: remove by age, wipe periodically, or move to remote storage Relevant docs: - Prefect results documentation - Prefect caching documentation - Settings and profiles If you want, I can also give you: - a safe cleanup script for
~/.prefect/storage
, or - guidance on switching local result storage to S3/GCS/Azure.
y
@Marvin how is the housekeeping works here?
m
thought for 10.2 seconds
There isn’t really any automatic housekeeping for
~/.prefect/storage
in Prefect itself. How housekeeping works - Prefect writes result files there when local result persistence is used - Prefect reads them later for cached results / persisted results - Prefect does not automatically prune old files from that directory - If a result has an expiration configured, Prefect can treat it as expired for usage purposes, but that does not mean the file is automatically deleted from disk So in practice, housekeeping is usually your responsibility: - periodic deletion by age - deleting expired results - wiping the directory when you no longer need local results - moving to remote storage with its own lifecycle policy What Prefect does with expired results Conceptually: - Prefect stores metadata including expiration when applicable - when checking whether a cached/persisted result is valid, expired results are no longer considered usable - but the underlying file may still remain on disk until you remove it So “housekeeping” in Prefect is more like: - logical invalidation of expired results - not physical cleanup of files A typical housekeeping approach For local disk, most people do one of these: 1. Delete files older than N days
Copy code
bash
find ~/.prefect/storage -type f -mtime +30 -delete
2. Run a cron job/systemd timer Example cron entry to clean files older than 14 days nightly:
Copy code
bash
0 2 * * * find /home/<user>/.prefect/storage -type f -mtime +14 -delete
3. Delete only when no runs are active If this machine is also running workers/flows, clean during a maintenance window. 4. Use remote object storage instead Then housekeeping can be managed with bucket lifecycle rules. Rule of thumb - If you rely on local caching only for short-term acceleration, age-based deletion is fine - If you rely on persisted results for recovery/retries, be more conservative - If runs are distributed or long-lived, prefer remote result storage If you want, I can give you a recommended cron job / cleanup script depending on whether this is: - a local dev machine - a VM running workers - Docker/Kubernetes-based execution