Shubham
05/28/2026, 6:42 AMMarvin
05/28/2026, 6:42 AMMarvin
05/28/2026, 6:44 AMbash
du -sh /* 2>/dev/null | sort -rh | head -20
Then specifically:
bash
du -sh ~/.prefect 2>/dev/null
du -sh ~/.prefect/storage 2>/dev/null
ls -lh ~/.prefect/prefect.db 2>/dev/null
du -sh /tmp 2>/dev/null
du -sh /var/lib/docker 2>/dev/null
What these are:
- ~/.prefect — Prefect home dir
- ~/.prefect/storage — local persisted results
- ~/.prefect/prefect.db — local SQLite DB if you’re not using Postgres
- /tmp — temp dirs from process workers / runs
- /var/lib/docker — very common culprit if this node runs Docker-based work
Most likely causes:
- Docker images/containers/volumes in /var/lib/docker
- Large local result storage in ~/.prefect/storage
- Large local SQLite DB in ~/.prefect/prefect.db
- General temp file buildup in /tmp
Useful follow-up commands:
bash
find / -type f -size +1G 2>/dev/null | head -100
find /tmp -type d -name '*prefect*' 2>/dev/null
docker system df 2>/dev/null
If you’re running a Prefect server/API node, also check your database storage. In Prefect 3, the DB can grow a lot from events/logs, especially the events table. Docs: Database maintenance
If you want, I can give you a safe cleanup checklist next depending on whether this node is:
- a worker node
- a server node
- or a Docker worker hostShubham
05/28/2026, 6:50 AMMarvin
05/28/2026, 6:50 AMMarvin
05/28/2026, 6:51 AM~/.prefect/prefect.db at 24 GB
2. Docker overlay storage under /var/lib/docker/... at ~98 GB — this is likely the immediate reason / filled up
Priority order: free Docker space first, then address prefect.db.
1) Docker: identify what is using space
Run:
bash
docker system df
docker ps -a --size
docker images
docker volume ls
du -sh /var/lib/docker/* 2>/dev/null | sort -rh | head -20
That overlay path means a container writable layer got huge. Find which container it is:
bash
docker inspect 65983772909c9775a405ec3016c0ca860712732092ba7632f02d8b69d9064500 \
--format '{{.Id}} {{.Name}}'
If that exact ID is not a container ID, map overlays to containers:
bash
docker ps -aq | while read c; do
echo "CONTAINER: $c $(docker inspect $c --format '{{.Name}}')";
docker inspect $c --format '{{.GraphDriver.Data.UpperDir}}' 2>/dev/null;
done | grep 65983772909c9775a405ec3016c0ca860712732092ba7632f02d8b69d9064500 -B1 -A1
2) Quick safe Docker cleanup
If you have stopped containers / dangling images / unused volumes:
bash
docker container prune -f
docker image prune -a -f
docker volume prune -f
Or all at once:
bash
docker system prune -a --volumes -f
Warning: this removes unused containers/images/volumes. Don’t run it if you rely on stopped containers or cached images staying around.
If the huge overlay belongs to a currently running flow container, inspect inside it for what filled the writable layer:
bash
docker exec -it <container_id> sh
du -sh /tmp /var/tmp /root /opt /app 2>/dev/null
find / -type f -size +1G 2>/dev/null | head -50
Common causes in Docker flow runs:
- files written to /tmp
- logs written inside container FS
- extracted datasets
- result files written locally instead of external storage
3) prefect.db at 24 GB
That means you’re using local SQLite on this node. For Prefect 3.x, that can happen if you’re running a local server/API with default DB settings. Likely growth is from events/logs/flow metadata.
Check size quickly:
bash
ls -lh ~/.prefect/prefect.db
sqlite3 ~/.prefect/prefect.db ".tables"
If you can use sqlite stats:
bash
sqlite3 ~/.prefect/prefect.db "SELECT name FROM sqlite_master WHERE type='table';"
sqlite3 ~/.prefect/prefect.db "VACUUM;"
Important: VACUUM rewrites the DB and needs free disk space roughly comparable to the DB size, so with / full it may fail.
A better long-term fix is:
- move Prefect server DB to Postgres
- reduce retention / clean up old data
- avoid running production-ish Prefect server on local SQLite
Docs: Database maintenance
4) Likely root cause pattern here
Given:
- Docker overlay is 98G
- Prefect DB is 24G
Your root partition was probably consumed by:
- one or more Docker flow containers writing huge local files
- plus accumulated Prefect metadata in SQLite
5) Recommended next actions
In this order:
bash
docker system df
docker ps -a --size
Then either:
- remove unused Docker artifacts, or
- identify the active container with the massive writable layer and inspect/delete its data
After you’ve freed some space:
- inspect prefect.db
- consider moving away from SQLite if this is a persistent Prefect server node
If you want, I can give you a very specific step-by-step cleanup sequence that minimizes risk on a live Docker/Prefect node.