<@ULVA73B9P> Sometime in the UI, my tasks get stuc...
# ask-marvin
c
@Marvin Sometime in the UI, my tasks get stuck in "pending" state. What does that mean and how do I fix this?
m
thought for 275.1 seconds
Short answer: “Pending” means the task run exists but hasn’t started executing yet. In Prefect 3.x, tasks commonly sit in Pending while they’re waiting on prerequisites (like upstream tasks), waiting for the flow’s task runner to have a free worker, or when the flow crashed/ended before the engine could schedule them. What typically causes tasks to stay in Pending - Waiting on upstreams: The task won’t start until all dependencies complete. Click the task in the UI to see the state’s message and check the graph for upstreams still Running/Retrying/Failed. - Task runner saturation: Inside a running flow, tasks run via your task runner. If there are more tasks than available worker slots, extras remain Pending until a worker frees up. - Deadlock-like patterns: If tasks block waiting for other tasks’ results while all workers are occupied, nothing can progress and tasks stay Pending. - Concurrency limits (by tag): If you’re using global concurrency limits, tasks may be in “AwaitingConcurrencySlot” until a slot opens. In some list views this can look like “pending” until you open the run details. - Flow crashed or exited early: If the flow failed/crashed before the engine scheduled the task, it will remain Pending indefinitely. How to fix it 1) Inspect the exact state and message - Open the task run in the UI and expand its state details. This usually tells you if it’s waiting on upstreams, a concurrency slot, etc. - States reference: States in Prefect 2) Ensure upstreams are completing - If an upstream failed or is still running/retrying, that’s expected. Resolve the upstream issue first. 3) Check for tag-based concurrency limits - List limits and see active slots:
Copy code
prefect concurrency-limit ls
  prefect concurrency-limit inspect <tag>
- If you need more capacity, increase or reset the limit:
Copy code
prefect concurrency-limit create <tag> <new_limit>
  prefect concurrency-limit reset <tag>
- Docs: Global concurrency limits 4) Increase task-runner capacity or change runner - By default, flows use a thread-based task runner. If you’re saturating workers, bump
max_workers
or switch runners:
Copy code
from prefect import flow
  from prefect.task_runners import ThreadPoolTaskRunner, ProcessPoolTaskRunner

  @flow(task_runner=ThreadPoolTaskRunner(max_workers=16))
  def my_flow():
      ...

  # For CPU-bound workloads:
  @flow(task_runner=ProcessPoolTaskRunner(max_workers=4))
  def cpu_flow():
      ...
- Redeploy and re-run your flow after changing the task runner. 5) Avoid task-to-task blocking - Try not to call
.result()
or otherwise wait on other tasks from inside a task—do that in the flow where possible. This prevents worker threads from getting stuck waiting on work that can’t start. 6) If the flow crashed/ended - Pending tasks from a terminated flow will never start. Re-run the flow after fixing the root cause (check flow logs for exceptions). If you’re using a distributed task runner (e.g., prefect-dask), also ensure the cluster/scheduler has available workers and is reachable. If you can share a bit more, I can be more specific: - Are these tasks within a single flow run that’s still Running, or did the flow fail/finish? - Do the task state details say “AwaitingConcurrencySlot” or “waiting for upstream tasks”? - What task runner are you using and what’s
max_workers
? - Any concurrency limits configured on tags?
c
@Marvin after inspecting the logs, it looks like the task completed successfully, but the UI isn't properly displayed. How can I fix this?
m
thought for 159.8 seconds
Got it — if the logs show the task finished but the UI looks wrong, it’s usually a UI/refresh/cache desync rather than the run actually being stuck. Here’s how to confirm and fix: 1) Verify the backend state (not the UI) - Find/inspect the task run:
Copy code
prefect task-run ls --limit 50 --state-type COMPLETED
  prefect task-run inspect <task_run_id>
  prefect task-run logs <task_run_id>
- If the backend shows Completed, the UI is just stale. 2) Force the UI to refresh - Hard refresh the page (Shift+Reload). - Log out/in or open the run in an incognito window. - Clear site data/local storage for your Prefect domain (app.prefect.cloud or your self-hosted domain). - Try a different browser to rule out extensions/caching. 3) If the backend still says Pending but you see “success” logs - Look in flow/worker logs right after the task’s last log for any “failed to update state” or API connectivity errors. - Ensure your worker/flow process has stable network access to the API and the correct
PREFECT_API_URL
. - Retry on the latest Prefect 3.x patch in case this is a fixed race-condition:
Copy code
pip install -U prefect
Release notes: Latest Prefect releases 4) Self-hosted Server specifics - Make sure API and UI come from the same Prefect version. - Restart the server stack to clear any stale UI caches. - Check browser devtools Network/Console for request or websocket errors to your API. If you share: - Cloud or self-hosted? - Prefect version (
prefect --version
) - A task/flow run ID and a screenshot of the UI state details …I can dig in further. States reference for context: Prefect states