Chad LeJeune
12/31/2025, 4:19 PMMarvin
12/31/2025, 4:19 PMMarvin
12/31/2025, 4:22 PM/tmp fills up you’ll see:
OSError: [Errno 28] No space left on device: '/tmp/...prefect'
Key points:
- In Prefect 3.x there is no PREFECT_TEMP_DIR. Set the standard OS env var TMPDIR to move Prefect’s temp work to a larger path.
- Common causes: large repos downloaded by flow.from_source(...), many concurrent runs, small ephemeral storage quotas (Kubernetes), or small container root filesystem (Docker).
Quick triage
1) Check and free space
- On the worker host/container:
df -h /tmp
du -sh /tmp
find /tmp -maxdepth 2 -name '*prefect*' -type d -exec du -sh {} \;
- Clean stale temp dirs (safe; Prefect recreates them):
rm -rf /tmp/runner_storage* /tmp/tmp*prefect
- Restart the worker after cleanup.
2) Point temp to a larger volume
- Local/VM (bash before starting the worker):
export TMPDIR=/mnt/large-disk/tmp
mkdir -p "$TMPDIR"
- Docker:
docker run \
-e TMPDIR=/mnt/large-disk/tmp \
-v /path/on/host:/mnt/large-disk \
prefecthq/prefect:3-latest
- Kubernetes (mount storage and set TMPDIR):
containers:
- name: prefect-worker
image: prefecthq/prefect:3-latest
env:
- name: TMPDIR
value: /mnt/large-tmp
volumeMounts:
- name: large-tmp
mountPath: /mnt/large-tmp
volumes:
- name: large-tmp
emptyDir:
sizeLimit: 100Gi
Or mount a PVC instead of emptyDir if you prefer.
3) Increase ephemeral storage limits (Kubernetes)
resources:
requests:
ephemeral-storage: "50Gi"
limits:
ephemeral-storage: "100Gi"
4) Reduce temp usage
- Pre-bake your flow code and dependencies into the image instead of cloning large repos at runtime.
- Keep repos/deploy artifacts lean; avoid cloning the entire monorepo if only a subdir is needed.
- Limit concurrency if many runs are starting at once (each run may download code into temp).
Verify the fix
- Inside the worker, confirm what temp dir Prefect will use:
python -c "import tempfile, os; print('tempdir=', tempfile.gettempdir(), 'TMPDIR=', os.getenv('TMPDIR'))"
- Monitor space:
watch -n 5 'df -h /tmp && du -sh /tmp/runner_storage* 2>/dev/null'
If you can share a bit more about your setup I can give exact steps:
- Where is the worker running (Kubernetes, Docker, VM)?
- Which worker type are you using?
- Rough size of your repo/artifacts?
- Do you see this immediately on start, or only when a run begins?Chad LeJeune
01/01/2026, 3:51 AMMarvin
01/01/2026, 3:51 AMMarvin
01/01/2026, 3:55 AM~/.prefect/storage (on your worker that’s /home/prefect/.prefect/storage) is where Prefect persists flow/task results and caches when persistence is enabled. It’s controlled by the PREFECT_RESULTS_LOCAL_STORAGE_PATH setting. There’s no automatic cleanup/retention; files accumulate until you delete them or move storage elsewhere.
Why it filled up
- Some flows/tasks are writing large results (e.g., big DataFrames or lists) with persist_result=True or you’ve enabled global persistence.
- You may also have set a default result storage block pointing to local disk.
- Caching can contribute too if results are frequently written and retained.
How to keep it from filling up
Option A: Turn off result persistence where you don’t need it
- Globally (default is already False, but verify):
prefect config view | grep PREFECT_RESULTS_PERSIST_BY_DEFAULT
prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT=false
- Per flow/task:
from prefect import flow, task
@task(persist_result=False)
def heavy_task():
...
@flow(persist_result=False)
def my_flow():
...
Option B: Redirect local result storage to a larger volume
- Point the setting to a mounted path with more space:
prefect config set PREFECT_RESULTS_LOCAL_STORAGE_PATH=/mnt/large-results/prefect-storage
- Ensure that path exists and is writable by the worker user.
Option C: Use remote object storage (S3/GCS/Azure) for results
- Create and save a storage block (example: S3, requires prefect-aws installed in your worker image/env):
from prefect_aws.s3 import S3Bucket
S3Bucket(bucket_name="my-bucket", basepath="prefect-results").save(
"results-s3", overwrite=True
)
- Make it the default for results:
prefect config set PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK='s3-bucket/results-s3'
- Or set per flow/task:
from prefect import flow
@flow(result_storage='s3-bucket/results-s3')
def my_flow():
...
Option D: Reduce size of stored results if you must persist them
- Use compressed serializers:
from prefect import flow
from prefect.serializers import CompressedPickleSerializer
@flow(persist_result=True, result_serializer=CompressedPickleSerializer())
def my_flow():
...
- Persist only what you need; avoid persisting very large objects.
Clean up what’s already there (safe if you don’t need those historical results anymore)
- Delete all results:
rm -rf /home/prefect/.prefect/storage/*
- Or prune older than 30 days:
find /home/prefect/.prefect/storage -type f -mtime +30 -delete
- Automate with cron:
0 2 * * 0 find /home/prefect/.prefect/storage -type f -mtime +30 -delete
Note: Deleting files for recent/running flows can cause “result missing” if something tries to read them later. Prefer pruning older data or after runs are well out of retry windows.
Useful checks
- See current settings:
prefect config view | egrep 'PREFECT_RESULTS_LOCAL_STORAGE_PATH|PREFECT_RESULTS_PERSIST_BY_DEFAULT|PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK'
- See what’s taking space:
du -sh /home/prefect/.prefect/storage
ls -lhS /home/prefect/.prefect/storage | head -n 50
If you share how the worker is running (Docker, Kubernetes, VM) and whether you need long-term result access, I can give exact, environment-specific steps (e.g., mount a bigger volume in your pod or container and set PREFECT_RESULTS_LOCAL_STORAGE_PATH accordingly).Chad LeJeune
01/01/2026, 4:19 AMMarvin
01/01/2026, 4:19 AMMarvin
01/01/2026, 4:20 AM~/.prefect/storage (i.e., PREFECT_RESULTS_LOCAL_STORAGE_PATH). Files written there will accumulate until you remove them or point results somewhere else.
Ways to prevent future buildup
- Don’t persist results unless needed
- Globally:
prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT=false
- Per flow/task:
@flow(persist_result=False)
def my_flow(): ...
@task(persist_result=False)
def my_task(): ...
- Move result storage to a bucket with lifecycle rules (recommended)
- Create a storage block (e.g., S3/GCS/Azure) and set it as default or per flow:
prefect config set PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK='s3-bucket/<your-block-name>'
Then enforce retention with the cloud provider’s lifecycle policy.
- Redirect local results to a larger, managed path
prefect config set PREFECT_RESULTS_LOCAL_STORAGE_PATH=/mnt/large-results/prefect-storage
Add your own cleanup (if you must keep local)
- Time-based prune (safe if the TTL exceeds your max retry/inspection windows):
find /home/prefect/.prefect/storage -type f -mtime +30 -delete
Add as a cron job or run via a small Prefect “housekeeping” deployment on a schedule.
- Size-capped prune (delete oldest until under a threshold):
#!/usr/bin/env bash
DIR=/home/prefect/.prefect/storage
LIMIT_BYTES=$((50 * 1024 * 1024 * 1024)) # 50GB
usage=$(du -sb "$DIR" | cut -f1)
if [ "$usage" -gt "$LIMIT_BYTES" ]; then
ls -1t "$DIR" | tail -n +1000 | while read f; do
rm -f "$DIR/$f"
usage=$(du -sb "$DIR" | cut -f1)
[ "$usage" -le "$LIMIT_BYTES" ] && break
done
fi
Kubernetes/Docker tips
- Kubernetes: if you want automatic reclamation on restart, point PREFECT_RESULTS_LOCAL_STORAGE_PATH to an emptyDir mount with a `sizeLimit`; the data will be cleared when the pod restarts.
- Docker: mount a host volume and use system cron on the host to prune, or move to object storage as above.
Caveat
- Deleting very recent files can break result reads for in‑flight or just-finished runs. Choose TTLs larger than your max retry/late-read windows.
If you share where your worker runs (Docker/K8s/VM) and whether you need long-term result access, I can give a precise cleanup or migration plan.Chad LeJeune
01/01/2026, 4:21 AMMarvin
01/01/2026, 4:21 AMMarvin
01/01/2026, 4:23 AMpersist_result=True on a task/flow, or when PREFECT_RESULTS_PERSIST_BY_DEFAULT=true, Prefect serializes the return value (pickle by default) and writes it to this directory using the LocalFileSystem block.
- Task scheduling data for autonomous/deferred tasks
- When tasks are served or run autonomously, their parameters and small scheduling artifacts are stored using the “task scheduling storage.” By default this also points to the same local path unless you set a dedicated block.
- Optional cache-policy metadata (if you configure a cache policy that writes to disk)
- If you set a cache policy with a file-based key storage, it will persist keys/metadata. If you do not, cache keys may be colocated with results.
What is NOT stored there
- The in-process LRU cache that speeds up repeated reads during a run is memory-only; it does not write to disk.
- Runner temporary code checkout/packaging is in the OS temp dir (e.g., /tmp/...), not in ~/.prefect/storage.
- Memoization for server internals uses $PREFECT_HOME/memo_store.toml, not the storage directory.
How to reduce or avoid growth
- Don’t persist results unless required:
prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT=false
and/or set persist_result=False on specific flows/tasks.
- Point result storage to object storage with lifecycle rules (recommended) and/or a larger path:
prefect config set PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK='s3-bucket/<your-block>'
prefect config set PREFECT_RESULTS_LOCAL_STORAGE_PATH=/mnt/large/results
- If you use autonomous/deferred tasks, set a separate storage block for scheduling data:
prefect config set PREFECT_TASKS_SCHEDULING_DEFAULT_STORAGE_BLOCK='<block-type>/<block-name>'
- If you must keep local, set up a prune job:
find /home/prefect/.prefect/storage -type f -mtime +30 -delete
If you tell me whether you’re persisting results intentionally and whether you use served/autonomous tasks, I can suggest precise settings to move only the necessary parts off local disk.