<@ULVA73B9P> How do I resolve the error: OSError: ...
# ask-marvin
c
@Marvin How do I resolve the error: OSError: [Errno 28] No space left on device: '/tmp/tmpl3o7ultkprefect' I cant submit flow runs to my infrastructure.
m
thought for 153.9 seconds
That error means the worker’s temp filesystem is full. Prefect uses the OS temp dir (via Python’s tempfile) for things like downloading source and packaging runs, so when
/tmp
fills up you’ll see: OSError: [Errno 28] No space left on device: '/tmp/...prefect' Key points: - In Prefect 3.x there is no
PREFECT_TEMP_DIR
. Set the standard OS env var
TMPDIR
to move Prefect’s temp work to a larger path. - Common causes: large repos downloaded by
flow.from_source(...)
, many concurrent runs, small ephemeral storage quotas (Kubernetes), or small container root filesystem (Docker). Quick triage 1) Check and free space - On the worker host/container:
Copy code
df -h /tmp
du -sh /tmp
find /tmp -maxdepth 2 -name '*prefect*' -type d -exec du -sh {} \;
- Clean stale temp dirs (safe; Prefect recreates them):
Copy code
rm -rf /tmp/runner_storage* /tmp/tmp*prefect
- Restart the worker after cleanup. 2) Point temp to a larger volume - Local/VM (bash before starting the worker):
Copy code
export TMPDIR=/mnt/large-disk/tmp
mkdir -p "$TMPDIR"
- Docker:
Copy code
docker run \
  -e TMPDIR=/mnt/large-disk/tmp \
  -v /path/on/host:/mnt/large-disk \
  prefecthq/prefect:3-latest
- Kubernetes (mount storage and set TMPDIR):
Copy code
containers:
- name: prefect-worker
  image: prefecthq/prefect:3-latest
  env:
  - name: TMPDIR
    value: /mnt/large-tmp
  volumeMounts:
  - name: large-tmp
    mountPath: /mnt/large-tmp
volumes:
- name: large-tmp
  emptyDir:
    sizeLimit: 100Gi
Or mount a PVC instead of emptyDir if you prefer. 3) Increase ephemeral storage limits (Kubernetes)
Copy code
resources:
  requests:
    ephemeral-storage: "50Gi"
  limits:
    ephemeral-storage: "100Gi"
4) Reduce temp usage - Pre-bake your flow code and dependencies into the image instead of cloning large repos at runtime. - Keep repos/deploy artifacts lean; avoid cloning the entire monorepo if only a subdir is needed. - Limit concurrency if many runs are starting at once (each run may download code into temp). Verify the fix - Inside the worker, confirm what temp dir Prefect will use:
Copy code
python -c "import tempfile, os; print('tempdir=', tempfile.gettempdir(), 'TMPDIR=', os.getenv('TMPDIR'))"
- Monitor space:
Copy code
watch -n 5 'df -h /tmp && du -sh /tmp/runner_storage* 2>/dev/null'
If you can share a bit more about your setup I can give exact steps: - Where is the worker running (Kubernetes, Docker, VM)? - Which worker type are you using? - Rough size of your repo/artifacts? - Do you see this immediately on start, or only when a run begins?
c
@Marvin The issue is filled /home/prefect/.prefect/storage on the worker. What does this directory do and how do I make sure is doesn't fill up in the future?
m
thought for 246.0 seconds
Great question — that path is Prefect’s default local result storage. What it is - In Prefect 3.x,
~/.prefect/storage
(on your worker that’s
/home/prefect/.prefect/storage
) is where Prefect persists flow/task results and caches when persistence is enabled. It’s controlled by the
PREFECT_RESULTS_LOCAL_STORAGE_PATH
setting. There’s no automatic cleanup/retention; files accumulate until you delete them or move storage elsewhere. Why it filled up - Some flows/tasks are writing large results (e.g., big DataFrames or lists) with
persist_result=True
or you’ve enabled global persistence. - You may also have set a default result storage block pointing to local disk. - Caching can contribute too if results are frequently written and retained. How to keep it from filling up Option A: Turn off result persistence where you don’t need it - Globally (default is already False, but verify):
Copy code
prefect config view | grep PREFECT_RESULTS_PERSIST_BY_DEFAULT
prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT=false
- Per flow/task:
Copy code
from prefect import flow, task

@task(persist_result=False)
def heavy_task():
    ...

@flow(persist_result=False)
def my_flow():
    ...
Option B: Redirect local result storage to a larger volume - Point the setting to a mounted path with more space:
Copy code
prefect config set PREFECT_RESULTS_LOCAL_STORAGE_PATH=/mnt/large-results/prefect-storage
- Ensure that path exists and is writable by the worker user. Option C: Use remote object storage (S3/GCS/Azure) for results - Create and save a storage block (example: S3, requires prefect-aws installed in your worker image/env):
Copy code
from prefect_aws.s3 import S3Bucket

S3Bucket(bucket_name="my-bucket", basepath="prefect-results").save(
    "results-s3", overwrite=True
)
- Make it the default for results:
Copy code
prefect config set PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK='s3-bucket/results-s3'
- Or set per flow/task:
Copy code
from prefect import flow

@flow(result_storage='s3-bucket/results-s3')
def my_flow():
    ...
Option D: Reduce size of stored results if you must persist them - Use compressed serializers:
Copy code
from prefect import flow
from prefect.serializers import CompressedPickleSerializer

@flow(persist_result=True, result_serializer=CompressedPickleSerializer())
def my_flow():
    ...
- Persist only what you need; avoid persisting very large objects. Clean up what’s already there (safe if you don’t need those historical results anymore) - Delete all results:
Copy code
rm -rf /home/prefect/.prefect/storage/*
- Or prune older than 30 days:
Copy code
find /home/prefect/.prefect/storage -type f -mtime +30 -delete
- Automate with cron:
Copy code
0 2 * * 0 find /home/prefect/.prefect/storage -type f -mtime +30 -delete
Note: Deleting files for recent/running flows can cause “result missing” if something tries to read them later. Prefer pruning older data or after runs are well out of retry windows. Useful checks - See current settings:
Copy code
prefect config view | egrep 'PREFECT_RESULTS_LOCAL_STORAGE_PATH|PREFECT_RESULTS_PERSIST_BY_DEFAULT|PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK'
- See what’s taking space:
Copy code
du -sh /home/prefect/.prefect/storage
ls -lhS /home/prefect/.prefect/storage | head -n 50
If you share how the worker is running (Docker, Kubernetes, VM) and whether you need long-term result access, I can give exact, environment-specific steps (e.g., mount a bigger volume in your pod or container and set
PREFECT_RESULTS_LOCAL_STORAGE_PATH
accordingly).
c
@Marvin Can I configure an expiration/cleanup for this data in prefect?
m
thought for 26.3 seconds
Short answer: not natively. Prefect 3.x does not provide a built‑in TTL/retention policy for
~/.prefect/storage
(i.e.,
PREFECT_RESULTS_LOCAL_STORAGE_PATH
). Files written there will accumulate until you remove them or point results somewhere else. Ways to prevent future buildup - Don’t persist results unless needed - Globally:
Copy code
prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT=false
- Per flow/task:
Copy code
@flow(persist_result=False)
  def my_flow(): ...
  @task(persist_result=False)
  def my_task(): ...
- Move result storage to a bucket with lifecycle rules (recommended) - Create a storage block (e.g., S3/GCS/Azure) and set it as default or per flow:
Copy code
prefect config set PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK='s3-bucket/<your-block-name>'
Then enforce retention with the cloud provider’s lifecycle policy. - Redirect local results to a larger, managed path
Copy code
prefect config set PREFECT_RESULTS_LOCAL_STORAGE_PATH=/mnt/large-results/prefect-storage
Add your own cleanup (if you must keep local) - Time-based prune (safe if the TTL exceeds your max retry/inspection windows):
Copy code
find /home/prefect/.prefect/storage -type f -mtime +30 -delete
Add as a cron job or run via a small Prefect “housekeeping” deployment on a schedule. - Size-capped prune (delete oldest until under a threshold):
Copy code
#!/usr/bin/env bash
  DIR=/home/prefect/.prefect/storage
  LIMIT_BYTES=$((50 * 1024 * 1024 * 1024))  # 50GB
  usage=$(du -sb "$DIR" | cut -f1)
  if [ "$usage" -gt "$LIMIT_BYTES" ]; then
    ls -1t "$DIR" | tail -n +1000 | while read f; do
      rm -f "$DIR/$f"
      usage=$(du -sb "$DIR" | cut -f1)
      [ "$usage" -le "$LIMIT_BYTES" ] && break
    done
  fi
Kubernetes/Docker tips - Kubernetes: if you want automatic reclamation on restart, point
PREFECT_RESULTS_LOCAL_STORAGE_PATH
to an
emptyDir
mount with a `sizeLimit`; the data will be cleared when the pod restarts. - Docker: mount a host volume and use system cron on the host to prune, or move to object storage as above. Caveat - Deleting very recent files can break result reads for in‑flight or just-finished runs. Choose TTLs larger than your max retry/late-read windows. If you share where your worker runs (Docker/K8s/VM) and whether you need long-term result access, I can give a precise cleanup or migration plan.
c
@Marvin What is stored there? Is it just the cache?
m
thought for 100.4 seconds
Short answer: it’s not “just the cache.” What’s stored in /home/prefect/.prefect/storage (Prefect 3.x) - Persisted task and flow results - When
persist_result=True
on a task/flow, or when
PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
, Prefect serializes the return value (pickle by default) and writes it to this directory using the LocalFileSystem block. - Task scheduling data for autonomous/deferred tasks - When tasks are served or run autonomously, their parameters and small scheduling artifacts are stored using the “task scheduling storage.” By default this also points to the same local path unless you set a dedicated block. - Optional cache-policy metadata (if you configure a cache policy that writes to disk) - If you set a cache policy with a file-based key storage, it will persist keys/metadata. If you do not, cache keys may be colocated with results. What is NOT stored there - The in-process LRU cache that speeds up repeated reads during a run is memory-only; it does not write to disk. - Runner temporary code checkout/packaging is in the OS temp dir (e.g.,
/tmp/...
), not in
~/.prefect/storage
. - Memoization for server internals uses
$PREFECT_HOME/memo_store.toml
, not the storage directory. How to reduce or avoid growth - Don’t persist results unless required:
Copy code
prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT=false
and/or set
persist_result=False
on specific flows/tasks. - Point result storage to object storage with lifecycle rules (recommended) and/or a larger path:
Copy code
prefect config set PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK='s3-bucket/<your-block>'
prefect config set PREFECT_RESULTS_LOCAL_STORAGE_PATH=/mnt/large/results
- If you use autonomous/deferred tasks, set a separate storage block for scheduling data:
Copy code
prefect config set PREFECT_TASKS_SCHEDULING_DEFAULT_STORAGE_BLOCK='<block-type>/<block-name>'
- If you must keep local, set up a prune job:
Copy code
find /home/prefect/.prefect/storage -type f -mtime +30 -delete
If you tell me whether you’re persisting results intentionally and whether you use served/autonomous tasks, I can suggest precise settings to move only the necessary parts off local disk.