Short answer: In Prefect 3.x, Pending means the flow run has been claimed for execution but hasn’t started running yet. A worker has begun the submission step (provisioning/starting the run), but the run hasn’t transitioned to Running. What your logs mean - INFO submitting flow run ... → The process worker claimed the run and started the submission step. - DEBUG Skipping ... because it’s already being submitted → The worker’s poll loop saw the same run again while a submission is still in-flight; it skips to avoid duplicating work. This is protective and normal for a short time. - DEBUG Ready to submit ... True → The worker determined it’s allowed to submit the run (capacity/conditions met). If many runs stay Pending, it usually means the submission step is hanging or never completes on the worker. Most common causes for “stuck in Pending” with a process worker - The worker is mid‑submission but blocked (e.g., subprocess can’t start, import/code path issues, working directory problems, missing env/credentials). - Worker lost connectivity to the API or crashed mid‑submission and never completed the state transition. - Worker capacity or machine resource starvation (cannot spawn new processes). - Version or configuration mismatch between deployment and worker (e.g., code not available at runtime). Quick things to try 1) Restart the worker process This clears any in‑memory “already being submitted” bookkeeping and often lets runs proceed. - Stop the worker and start it again, ensuring it points at the correct work pool/queues:

prefect worker start --pool <your-work-pool> --work-queue <your-queue>

2) Enable debug logs on the worker You’ll get more detail about what’s blocking submission:

export PREFECT_LOGGING_LEVEL=DEBUG
prefect worker start --pool <your-work-pool> --work-queue <your-queue>

3) Verify the worker can actually start a subprocess for your deployment - Check that the working directory configured for the deployment is valid on the worker host. - If you’ve overridden the command or base job template, try running that command manually on the worker machine. - Confirm required environment variables/secrets are present on the worker. 4) Check the Prefect UI for one affected run - Open a Pending run → Details → State history. Look for any messages around “submitting” or failures during submission. Also check the run’s logs (if stream_output is on) for early process errors. 5) Confirm worker health and capacity - The worker should show Online in the Work Pools page. - Make sure it’s subscribed to the correct work queue(s) and isn’t at max capacity (limit) or starved for CPU/RAM. If it persists, please share: - Prefect version (server/Cloud and worker), OS of the worker host. - A single example run ID and its deployment config (working_dir, any custom command/base_job_template). - Whether a worker restart helped and any new DEBUG-level log lines after restart. References - States (incl. Pending definition): Prefect docs: States - Workers overview (incl. process worker): Prefect docs: Workers - Process worker API reference: API reference: Process worker - Troubleshooting stuck runs / worker issues: Troubleshoot runs and workers

Short answer: No—there isn’t a bulk “retry/resubmit all Pending runs” action in the Prefect 3.x UI today. What you can do instead - Fix the underlying cause and restart the worker. If the process worker was mid‑submission, a restart often clears Pending runs to proceed. - Reschedule Pending runs via the API/SDK. Example script to set all Pending runs back to Scheduled so workers can pick them up again (batch/target as needed):

import asyncio
from prefect.client.orchestration import get_client
from prefect.client.schemas.filters import (
    FlowRunFilter, FlowRunFilterState, FlowRunFilterStateType
)
from prefect.states import Scheduled

async def main():
    async with get_client() as client:
        # Adjust filters (e.g., by deployment, tags) to target the right runs
        runs = await client.read_flow_runs(
            flow_run_filter=FlowRunFilter(
                state=FlowRunFilterState(
                    type=FlowRunFilterStateType(any_=["PENDING"])
                )
            ),
            limit=500,  # tune batch size
        )
        for r in runs:
            await client.set_flow_run_state(
                r.id,
                Scheduled(message="Rescheduled from Pending"),
                force=True,  # override if a worker was mid-submission
            )

asyncio.run(main())

Notes: - There’s no CLI bulk retry for Pending;

prefect flow-run retry

only applies to terminal states (Completed/Failed/Cancelled/Crashed). - If a worker is actively submitting a run, forcing state changes can race—consider pausing/restarting the worker first or scoping the script narrowly (e.g., by deployment). If you want, tell me which deployment(s) or filters to use (deployment name, tags, work pool/queue), and I’ll tailor a safe, scoped script for your environment. Helpful docs - Troubleshooting stuck runs/workers: Troubleshoot runs and workers