<@ULVA73B9P> Can you help me figure out why my flo...
# ask-marvin
s
@Marvin Can you help me figure out why my flows have been in a pending state for the last 10 hours? I am not using any concurrency limits in these flows and I am using ECS
m
thought for 515.8 seconds
Got it — when flow runs sit in Pending for hours (especially with Docker), it almost always means the worker isn’t picking them up. In Prefect 3.x, Pending means “ready to run but waiting for infrastructure.” Common causes: - No worker is polling the work pool/queue (worker not running, wrong pool/queue, or lost API connection) - Work queue is paused or not READY (hasn’t been polled recently) - Tag/queue filters exclude your runs (deployment tags don’t match queue filters) - Hidden concurrency limits (global or queue-level) - Docker image issues (private registry auth, image not found, slow/failed pull) - Deployment points at a different pool/queue than your worker Quick checks you can run - List pools and verify status:
Copy code
prefect work-pool ls -v
- List queues in your pool and verify they’re not PAUSED and were polled recently:
Copy code
prefect work-queue ls -p <your-pool-name> -v
prefect work-queue inspect <your-queue-name> -p <your-pool-name>
- Check for any concurrency limits:
Copy code
prefect concurrency-limit ls
- Inspect the deployment to confirm the work pool, tags, and Docker image:
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
- Start a worker (if one isn’t running) and watch logs:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG prefect worker start --pool <your-pool-name>
What to look for - In the UI, Work Pools page: is your pool READY (green)? Is the queue not PAUSED and showing a recent “last polled”? If not, the worker isn’t running or can’t reach the API. - In the worker logs: do you see “No work found” repeatedly? That indicates tag/queue filter mismatch or the worker is polling a different pool/queue. - In the flow run details: expand the current state to read the message — e.g., waiting on concurrency slot, NotReady, or infra messages. - Docker image: confirm the image in your deployment/job variables exists and can be pulled from the worker host:
Copy code
docker pull <image-tag>
- API connectivity: if using Cloud, ensure
PREFECT_API_URL
and
PREFECT_API_KEY
are set in the worker environment. If self-hosted, ensure the worker can reach your server URL. Helpful docs - States (Pending vs Scheduled) - Work pools and queues - Workers overview - Docker deployments - Concurrency limits To help you faster, could you share: - Prefect Cloud or self-hosted? - The work pool and work queue names your deployment targets - Whether a worker is currently running for that pool (and any recent worker log lines) - The output of:
Copy code
prefect work-pool ls -v
prefect work-queue ls -p <your-pool-name> -v
prefect deployment inspect "<flow-name>/<deployment-name>"
prefect concurrency-limit ls
- The Docker image set for the deployment and whether
docker pull
succeeds on the worker host With those details, I can pinpoint why your runs are stuck in Pending.