< Marvin> what to do in this situation where runs in status Prefect Community #ask-marvin

<@ULVA73B9P> what to do in this situation where. r...

Shubham

08/08/2026, 6:21 AM

@Marvin what to do in this situation where. runs in status submitting from last 30 min each new run is stucking on submitting status.

Marvin

08/08/2026, 6:21 AM

✅ thought for 63.1 seconds

Marvin

08/08/2026, 6:22 AM

If every flow run is stuck in Submitting, Prefect has accepted the run, but it is not successfully handing it off to infrastructure yet. Most common causes are: 1. No worker is polling the work pool 2. Worker is unhealthy / disconnected 3. Worker picked up the run but cannot create the job - bad Docker/Kubernetes credentials - image pull failures - missing env vars / secrets - invalid job template 4. Concurrency / pool limits blocking submission 5. Temporary API / infrastructure issue What “Submitting” usually means - The run was created - Prefect is trying to submit it to your work pool’s infrastructure - It has not started running yet Things to check first 1. Check whether your worker is online Go to: - Work Pools in the UI and open the pool used by this deployment - Confirm there is a healthy worker attached If there is no active worker, runs will sit and wait. If you use CLI, check worker commands first:

Copy code

bash
prefect worker --help

Then make sure a worker is running for the correct pool, for example:

Copy code

bash
prefect worker start --pool <your-pool-name>

2. Check the flow run page events/logs Open one of the stuck runs and look for: - event messages - infrastructure errors - worker submission errors Typical messages: - failed to create job - image pull backoff - authentication failure - Kubernetes forbidden / unauthorized - Docker daemon unavailable 3. Check worker logs directly Wherever your worker process is running, inspect its logs. You’re looking for errors like: - cannot connect to Docker daemon - cannot create Kubernetes job - invalid work pool job variables - network/authentication errors to Prefect API 4. Check your work pool type The fix depends on the pool type: - Process pool: machine running worker may be overloaded or unable to spawn subprocesses - Docker pool: Docker daemon/image issues - Kubernetes pool: RBAC, namespace, service account, image pull, cluster connectivity - Cloud-run/serverless style pool: cloud credentials or quota issues 5. Check concurrency limits If you have: - deployment concurrency limits - tag-based concurrency limits - global concurrency limits then runs can be delayed. Usually that shows as waiting, but it’s still worth checking if submission is blocked by limits. 6. Check if this started suddenly If it worked before and all new runs are now stuck: - worker may have stopped - worker lost API connectivity - cloud credentials expired - infrastructure backend is failing - recent deployment/job template change introduced invalid config Quick recovery steps Try these in order: 1. Restart the worker 2. Verify the worker is attached to the correct work pool 3. Run one manual test deployment/run 4. Check worker logs for the first error 5. Check infrastructure backend - Docker running? - Kubernetes cluster reachable? - credentials valid? 6. Pause/disable old bad runs if needed, then retry after fixing worker If you’re using Docker Check: - Docker daemon is up - image exists or can be pulled - registry credentials are valid - container can start on that host If you’re using Kubernetes Check: - worker pod is healthy - service account permissions - namespace exists - job/pod creation is allowed - image pull secret works - pods are not stuck in

Pending

Useful docs - Prefect workers: Workers - Work pools: Work Pools - Deployments: Deployments - Troubleshooting docs: Troubleshooting Most likely fix If all new runs are stuck for 30 minutes, the #1 likely cause is that your worker is not running or cannot submit jobs to the infrastructure backend. If you want, send me: - your work pool type (

process

docker

kubernetes

, etc.) - whether the worker is running

Marvin

08/08/2026, 6:22 AM

- any worker log error and I can help narrow it down quickly.

Shubham

08/08/2026, 6:23 AM

@Marvin root@Prefect:/home/prefect# df -h Filesystem Size Used Avail Use% Mounted on tmpfs 795M 1.1M 794M 1% /run /dev/sda2 98G 93G 203M 100% / tmpfs 3.9G 1.1M 3.9G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 795M 12K 795M 1% /run/user/1000 root@Prefect:/home/prefect#

Marvin

08/08/2026, 6:23 AM

✅ thought for 7.3 seconds

Marvin

08/08/2026, 6:23 AM

That’s very likely the problem. Your root disk is full:

Copy code

bash
/dev/sda2  98G  93G  203M  100% /

When the machine hosting the worker is out of disk, Prefect runs can get stuck in Submitting because the worker/infrastructure cannot create new execution environments, write temp files, pull images, create logs, or start subprocesses. What to do now 1. Free disk space immediately Start by finding what is consuming space:

Copy code

bash
du -xh / | sort -h | tail -50

A safer and more useful version:

Copy code

bash
du -sh /* 2>/dev/null

Then drill into the biggest directories, often: -

/var/log

/tmp

/home

/opt

/var/lib/docker

/root/.cache

/home/prefect/.cache

2. If you are using Docker, this is a very common culprit Check Docker usage:

Copy code

bash
docker system df

Clean up unused Docker resources:

Copy code

bash
docker system prune -a

If you also want to remove unused volumes:

Copy code

bash
docker system prune -a --volumes

Be careful: this removes unused images/containers/volumes. 3. Clear logs if they are huge Check log sizes:

Copy code

bash
du -sh /var/log/* 2>/dev/null | sort -h | tail -20

If a specific log is huge, truncate it:

Copy code

bash
truncate -s 0 /var/log/<large-log-file>

4. Clear temp/cache files Examples:

Copy code

bash
rm -rf /tmp/*
rm -rf /home/prefect/.cache/*
rm -rf /root/.cache/*

Only do this if you’re comfortable removing caches/temp files. 5. Restart the worker after freeing space Once you free a few GB at least, restart your Prefect worker. Then submit a new flow run and see whether it moves past Submitting. Recommended target Try to free at least several GB, not just a few hundred MB.

203M

free is critically low. Best next checks After cleanup, run:

Copy code

bash
df -h

and if using Docker:

Copy code

bash
docker system df

If you want, paste the output of:

Copy code

bash
du -sh /var/* /home/* 2>/dev/null | sort -h

and I can help identify the fastest/safest place to clean up.

Shubham

08/08/2026, 6:25 AM

@Marvin prefect=> SELECT pg_size_pretty(pg_database_size('prefect')) AS db_size; db_size --------- 2591 MB (1 row) prefect=> SELECT datname, pg_size_pretty(pg_database_size(datname)) AS size FROM pg_database ORDER BY pg_database_size(datname) DESC; datname | size -----------+--------- prefect | 2591 MB template1 | 7567 kB postgres | 7345 kB template0 | 7345 kB (4 rows) prefect=> SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname || '.' || tablename)) AS total_size FROM pg_tables WHERE schemaname = 'public' ORDER BY pg_total_relation_size(schemaname || '.' || tablename) DESC LIMIT 20; schemaname | tablename | total_size ------------+---------------------+------------ public | events | 848 MB public | event_resources | 815 MB public | task_run_state | 367 MB public | task_run | 235 MB public | log | 158 MB public | flow_run_state | 88 MB public | flow_run | 66 MB public | artifact | 3464 kB public | deployment | 432 kB public | block_schema | 304 kB public | automation_bucket | 280 kB public | block_type | 208 kB public | work_queue | 200 kB public | worker | 136 kB public | block_document | 136 kB public | work_pool | 128 kB public | flow | 104 kB public | deployment_schedule | 88 kB public | configuration | 80 kB public | variable | 64 kB (20 rows) prefect=>

Marvin

08/08/2026, 6:25 AM

✅ thought for 7.2 seconds

Marvin

08/08/2026, 6:25 AM

Your Prefect database is not the main disk problem. You showed: - DB size: ~2.5 GB - Root disk used: ~93 GB So Postgres is only a small part of the total usage. Even if you cleaned the DB completely, you’d still have a disk pressure problem elsewhere. That said, your largest Prefect tables are: -

events

→ 848 MB -

event_resources

→ 815 MB -

task_run_state

→ 367 MB -

task_run

→ 235 MB -

log

→ 158 MB So there is some recoverable space in Prefect metadata, but probably not enough to explain 93 GB used on

. Likely bigger disk consumers On a Prefect host, the usual large consumers are: - Docker images/containers/volumes:

/var/lib/docker

- system logs:

/var/log

- Python envs / caches - local result files - temp files - old artifacts or backups Next thing to check Please run this:

Copy code

bash
du -sh /var/* /home/* /root/* 2>/dev/null | sort -h

And if Docker is installed, also run:

Copy code

bash
du -sh /var/lib/docker 2>/dev/null
docker system df

That will probably reveal the real source quickly. About the Prefect DB specifically If you do want to reduce DB size, the biggest gain is usually from: - old

events

- old

logs

- old run/state history But be careful deleting directly from the DB unless you know your retention plan. If this is self-hosted Prefect Server, it’s generally safer to: 1. back up the database 2. delete old flow/task run history in a controlled way 3. vacuum afterward For PostgreSQL, deleting rows does not immediately return space to the OS. You usually need:

Copy code

sql
VACUUM (ANALYZE);

And sometimes more aggressive maintenance like

VACUUM FULL

on specific large tables, but that can lock tables. Important takeaway - Yes, low disk can absolutely cause runs to stick in

Submitting

- No, your Prefect DB alone is probably not the root of the 93 GB usage - Most likely culprit is still
/var/lib/docker
or logs If you paste the

du -sh /var/* /home/* /root/*

output, I can tell you exactly where to clean first.

Shubham

08/08/2026, 6:36 AM

@Marvin this is full root@Prefect:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs# ls full of directories why

Marvin

08/08/2026, 6:36 AM

✅ thought for 11.7 seconds

Marvin

08/08/2026, 6:36 AM

That explains it.

/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs

is where containerd stores container image layers and snapshots. If that directory is full of subdirectories and your disk is 100%, then the space is very likely being consumed by old/unreferenced container images, snapshots, or containers managed by containerd. This is consistent with Prefect runs getting stuck in Submitting if your worker launches containers. Why this happens Common reasons: - many image pulls over time - old container snapshots not cleaned up - repeated deployments using new image tags - failed/stuck containers leaving snapshots behind - Kubernetes / k3s / nerdctl / containerd-managed workloads accumulating layers Important: Don’t manually

rm -rf

random directories inside

/var/lib/containerd/...

That can corrupt containerd state. What to do instead 1. Confirm containerd usage Check whether this host is using: - Kubernetes - k3s - nerdctl - Docker backed by containerd - another runtime Useful checks:

Copy code

bash
ps aux | grep -E 'containerd|kubelet|k3s|docker'
systemctl status containerd
systemctl status docker

2. Measure the size Run:

Copy code

bash
du -sh /var/lib/containerd
du -sh /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs

3. If you use
nerdctl
, clean unused images/containers Check:

Copy code

bash
nerdctl system df

Then prune:

Copy code

bash
nerdctl system prune

And more aggressively if needed:

Copy code

bash
nerdctl system prune -a

4. If this is Kubernetes, check for old images For

crictl

environments:

Copy code

bash
crictl images
crictl ps -a

Remove unused images:

Copy code

bash
crictl rmi --prune

If there are exited containers/sandboxes:

Copy code

bash
crictl pods
crictl rm <container-id>
crictl rmp <pod-id>

5. If Docker is installed and actually in use Sometimes Docker and containerd both exist. Check Docker too:

Copy code

bash
docker system df
docker system prune -a

6. Restart the worker after cleanup Once you free enough disk:

Copy code

bash
df -h

Then restart your Prefect worker/infrastructure agent and try a new run. Very likely root cause for Prefect If Prefect is submitting runs to container-based infrastructure, and the node cannot create/pull/start containers because containerd storage is full, runs may remain in

Submitting

. Safe cleanup guidance Use the runtime tools, not direct file deletion: -

nerdctl ...

crictl ...

docker ...

not:

Copy code

bash
rm -rf /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/*

unless you are doing a full node rebuild and know exactly what depends on it. If you want the fastest next step, run these and paste the outputs:

Copy code

bash
du -sh /var/lib/containerd
du -sh /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
crictl images
crictl ps -a

crictl

is not installed, try:

Copy code

bash
nerdctl system df

Then I can tell you the safest cleanup command for your setup.

Shubham

08/08/2026, 6:37 AM

@Marvin im using docker worker on this host nothing else

Marvin

08/08/2026, 6:37 AM

✅ thought for 6.7 seconds

Marvin

08/08/2026, 6:37 AM

If you’re only using a Docker worker on this host, then the main thing to clean is usually Docker, not Prefect itself. But note: modern Docker uses containerd internally, so it’s normal to see lots of data under

/var/lib/containerd/...

. That space may still be from Docker-managed images/layers. Do this first 1. Check Docker disk usage

Copy code

bash
docker system df

This will show space used by: - images - containers - local volumes - build cache 2. Clean unused Docker resources Start with:

Copy code

bash
docker system prune

If you want a stronger cleanup:

Copy code

bash
docker system prune -a

And to also remove unused volumes:

Copy code

bash
docker system prune -a --volumes

Warning:

-a --volumes

can remove a lot. Only use it if you understand what’s safe to delete. 3. Check container list

Copy code

bash
docker ps -a

If you have many exited/stale containers, remove them:

Copy code

bash
docker container prune

4. Check image list

Copy code

bash
docker images

If you have many old images from deployments, remove unused ones:

Copy code

bash
docker image prune -a

5. Check volumes

Copy code

bash
docker volume ls

If there are many unused volumes:

Copy code

bash
docker volume prune

6. Recheck disk

Copy code

bash
df -h

Why this is affecting Prefect Your Docker worker needs disk space to: - pull images - create writable container layers - start containers - write logs/temp files With only ~203 MB free, Docker may fail to start new flow-run containers, so Prefect runs stay in Submitting. Recommended cleanup order If you want the fastest path:

Copy code

bash
docker system df
docker container prune -f
docker image prune -a -f
docker volume prune -f
docker builder prune -a -f
df -h

Or one-shot:

Copy code

bash
docker system prune -a --volumes -f

Be careful with volumes If any containers store important persistent data in Docker volumes, don’t prune volumes blindly. After cleanup 1. confirm several GB are free 2. restart Docker if needed:

Copy code

bash
systemctl restart docker

3. restart your Prefect worker 4. trigger a new run Likely outcome Once Docker has enough free space again, new runs should move past Submitting. If you want, paste the output of:

Copy code

bash
docker system df
docker ps -a
docker images

and I’ll tell you the safest cleanup command to run next.

Shubham

08/08/2026, 6:40 AM

@Marvin i have only two images root@Prefect:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots# docker images -a i Info → U In Use IMAGE ID DISK USAGE CONTENT SIZE EXTRA hello-world:latest 0e760fdfbc48 25.9kB 9.49kB U khuspeshubham/drx-elasticsearch:1.0 8668de464d1b 573MB 130MB U khuspeshubham/images:drx-securonix-v1 00e69266d3b6 1.15GB 285MB U <untagged> b216a643f0f9 428MB 13.1kB U root@Prefect:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots# this 2 deployments using this 2 images runs every 2 min. suggest me best way to prevent this in future and how to correct now

Marvin

08/08/2026, 6:40 AM

✅ thought for 11.2 seconds

Marvin

08/08/2026, 6:41 AM

You’re likely hitting Docker/containerd snapshot buildup from frequently created containers, not just large image count. Even with only 2–3 images, if your 2 deployments run every 2 minutes, Docker worker containers are being created constantly. If old containers/layers aren’t cleaned up fast enough, overlay/containerd storage can grow until the disk fills and runs get stuck in Submitting. What to do right now 1. See what is consuming Docker space Please run:

Copy code

bash
docker system df -v

That will show whether the real growth is in: - containers - images - local volumes - build cache 2. Remove stopped containers This is the most likely immediate cleanup:

Copy code

bash
docker container prune -f

3. Remove unused images/cache

Copy code

bash
docker image prune -a -f
docker builder prune -a -f

4. Remove unused volumes only if safe

Copy code

bash
docker volume ls
docker volume prune -f

Only do this if your flows are not storing needed data in named volumes. 5. Recheck free space

Copy code

bash
df -h
docker system df

6. Restart Docker and worker if needed

Copy code

bash
systemctl restart docker

Then restart your Prefect worker and test a new run. --- Why this happens even with only 2 images The problem is usually not image count. It’s one or more of: - many stopped containers accumulating - writable container layers from repeated runs - anonymous volumes - logs for many short-lived containers - failed run containers not being cleaned up - containers overlapping because runs happen every 2 minutes If each run creates a new container and the previous ones remain around, space usage grows over time. --- Best ways to prevent this in future 1. Add Docker cleanup on a schedule If this is a dedicated worker host, run periodic cleanup via cron. Example daily cleanup:

Copy code

bash
docker container prune -f
docker image prune -a -f
docker builder prune -a -f

If volumes are safe to remove:

Copy code

bash
docker volume prune -f

You can put that in cron, for example:

Copy code

bash
crontab -e

Add:

Copy code

bash
0 2 * * * docker container prune -f && docker image prune -a -f && docker builder prune -a -f

2. Monitor disk usage Set an alert when

exceeds 80–85%. Useful quick check:

Copy code

bash
df -h /

3. Use a larger disk If these deployments run every 2 minutes continuously, 98 GB may eventually be tight depending on: - log growth - temp files - output persistence - container churn 4. Check whether failed/stopped containers are accumulating Run:

Copy code

bash
docker ps -a

If you see a very large number of exited containers, that confirms the issue. 5. Keep image strategy stable Use fixed images and avoid rebuilding/pulling lots of unique tags unnecessarily. 6. Limit logs if container JSON logs are growing Check:

Copy code

bash
du -sh /var/lib/docker/containers 2>/dev/null

If this is large, Docker container logs may be growing. In that case, configure log rotation in Docker daemon. Typical `/etc/docker/daemon.json`:

Copy code

json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Then restart Docker:

Copy code

bash
systemctl restart docker

7. Consider reducing run frequency or overlap If both deployments run every 2 minutes and each run takes longer than 2 minutes, containers may stack up. Check whether runs overlap heavily: - if yes, use concurrency limits - or reduce schedule frequency - or make flows shorter/lighter --- Most useful checks right now Please run these:

Copy code

bash
docker system df -v
docker ps -a | wc -l
du -sh /var/lib/docker /var/lib/containerd /var/lib/docker/containers 2>/dev/null

That will tell us whether the main leak is: - stopped containers - logs - volumes - build cache - containerd snapshots --- My best guess from what you shared Because the deployments run every 2 minutes, the most likely root cause is: - many short-lived Docker containers accumulating

Marvin

08/08/2026, 6:41 AM

- possibly container logs/snapshots not being reclaimed quickly enough So the best immediate fix is:

Copy code

bash
docker container prune -f
docker image prune -a -f
docker builder prune -a -f
df -h

Then add scheduled cleanup and Docker log rotation. If you paste

docker system df -v

, I can tell you the exact prevention strategy for your host.

Open in Slack

Previous Next