<@ULVA73B9P> which file paths to check My / is ful...
# ask-marvin
s
@Marvin which file paths to check My / is full on prefect node
m
thought for 97.7 seconds
The main paths I’d check on a Prefect node are:
Copy code
bash
du -sh /* 2>/dev/null | sort -rh | head -20
Then specifically:
Copy code
bash
du -sh ~/.prefect 2>/dev/null
du -sh ~/.prefect/storage 2>/dev/null
ls -lh ~/.prefect/prefect.db 2>/dev/null
du -sh /tmp 2>/dev/null
du -sh /var/lib/docker 2>/dev/null
What these are: -
~/.prefect
— Prefect home dir -
~/.prefect/storage
— local persisted results -
~/.prefect/prefect.db
— local SQLite DB if you’re not using Postgres -
/tmp
— temp dirs from process workers / runs -
/var/lib/docker
— very common culprit if this node runs Docker-based work Most likely causes: - Docker images/containers/volumes in
/var/lib/docker
- Large local result storage in
~/.prefect/storage
- Large local SQLite DB in
~/.prefect/prefect.db
- General temp file buildup in
/tmp
Useful follow-up commands:
Copy code
bash
find / -type f -size +1G 2>/dev/null | head -100
find /tmp -type d -name '*prefect*' 2>/dev/null
docker system df 2>/dev/null
If you’re running a Prefect server/API node, also check your database storage. In Prefect 3, the DB can grow a lot from events/logs, especially the
events
table. Docs: Database maintenance If you want, I can give you a safe cleanup checklist next depending on whether this node is: - a worker node - a server node - or a Docker worker host
s
@Marvin Prefect.db is 24gb my disk was 100G . This node runs docker based flows And overlay 98G 98G 0 100% /var/lib/docker/rootfs/overlayfs/65983772909c9775a405ec3016c0ca860712732092ba7632f02d8b69d9064500
m
thought for 17.9 seconds
You’ve got two separate disk consumers: 1.
~/.prefect/prefect.db
at 24 GB 2. Docker overlay storage under
/var/lib/docker/...
at ~98 GB — this is likely the immediate reason
/
filled up
Priority order: free Docker space first, then address
prefect.db
. 1) Docker: identify what is using space Run:
Copy code
bash
docker system df
docker ps -a --size
docker images
docker volume ls
du -sh /var/lib/docker/* 2>/dev/null | sort -rh | head -20
That overlay path means a container writable layer got huge. Find which container it is:
Copy code
bash
docker inspect 65983772909c9775a405ec3016c0ca860712732092ba7632f02d8b69d9064500 \
  --format '{{.Id}} {{.Name}}'
If that exact ID is not a container ID, map overlays to containers:
Copy code
bash
docker ps -aq | while read c; do
  echo "CONTAINER: $c $(docker inspect $c --format '{{.Name}}')";
  docker inspect $c --format '{{.GraphDriver.Data.UpperDir}}' 2>/dev/null;
done | grep 65983772909c9775a405ec3016c0ca860712732092ba7632f02d8b69d9064500 -B1 -A1
2) Quick safe Docker cleanup If you have stopped containers / dangling images / unused volumes:
Copy code
bash
docker container prune -f
docker image prune -a -f
docker volume prune -f
Or all at once:
Copy code
bash
docker system prune -a --volumes -f
Warning: this removes unused containers/images/volumes. Don’t run it if you rely on stopped containers or cached images staying around. If the huge overlay belongs to a currently running flow container, inspect inside it for what filled the writable layer:
Copy code
bash
docker exec -it <container_id> sh
du -sh /tmp /var/tmp /root /opt /app 2>/dev/null
find / -type f -size +1G 2>/dev/null | head -50
Common causes in Docker flow runs: - files written to
/tmp
- logs written inside container FS - extracted datasets - result files written locally instead of external storage 3)
prefect.db
at 24 GB
That means you’re using local SQLite on this node. For Prefect 3.x, that can happen if you’re running a local server/API with default DB settings. Likely growth is from events/logs/flow metadata. Check size quickly:
Copy code
bash
ls -lh ~/.prefect/prefect.db
sqlite3 ~/.prefect/prefect.db ".tables"
If you can use sqlite stats:
Copy code
bash
sqlite3 ~/.prefect/prefect.db "SELECT name FROM sqlite_master WHERE type='table';"
sqlite3 ~/.prefect/prefect.db "VACUUM;"
Important:
VACUUM
rewrites the DB and needs free disk space roughly comparable to the DB size, so with
/
full it may fail. A better long-term fix is: - move Prefect server DB to Postgres - reduce retention / clean up old data - avoid running production-ish Prefect server on local SQLite Docs: Database maintenance 4) Likely root cause pattern here Given: - Docker overlay is 98G - Prefect DB is 24G Your root partition was probably consumed by: - one or more Docker flow containers writing huge local files - plus accumulated Prefect metadata in SQLite 5) Recommended next actions In this order:
Copy code
bash
docker system df
docker ps -a --size
Then either: - remove unused Docker artifacts, or - identify the active container with the massive writable layer and inspect/delete its data After you’ve freed some space: - inspect
prefect.db
- consider moving away from SQLite if this is a persistent Prefect server node If you want, I can give you a very specific step-by-step cleanup sequence that minimizes risk on a live Docker/Prefect node.