Thanks for the details — this does sound like it’s tied to worker availability and deployment status. Here’s how this fits together in Prefect Cloud (Prefect 3.x): - Deployment “ready/not-ready” events are emitted based on worker polling. When a worker polls your work pool/queue recently, you’ll see

prefect.deployment.ready

. If no workers have polled for a bit (about 60s), you’ll see

prefect.deployment.not-ready

. - Those events don’t stop the scheduler by themselves, but they correlate directly with runs not starting: if the deployment is “not-ready,” new runs will sit in Scheduled (often flip to Late) until a worker comes back and polls. If you have an Automation watching those events, it can also pause/resume things which will affect scheduling. Most common causes when runs don’t start or never complete - No worker is polling the exact work pool/queue the deployment uses (or the pool/queue is paused) - Concurrency limits are hit (deployment, pool, queue, or tag-based), so runs wait in

AwaitingConcurrencySlot

- Automation paused the deployment/pool/queue based on an event - Worker can’t access code/storage/image or fails provisioning infra (runs hang in Pending or crash early) - Long or missing heartbeats (runs appear “Running” but never complete; infra died and the run wasn’t marked) Quick checks you can run - Verify a stuck run’s state and history:

prefect flow-run ls
prefect flow-run inspect <flow-run-id>

- Confirm deployment configuration is pointing to the right pool/queue and is not paused, and that the schedule is active:

prefect deployment inspect <flow>/<deployment>

- Check pool/queue health and limits:

prefect work-pool ls
prefect work-pool inspect <pool-name>
prefect work-queue ls --pool <pool-name>

- Confirm workers are online and polling the same pool/queue:

prefect worker ls

- Look at the deployment’s Activity tab around the problem time for

prefect.deployment.not-ready

and

prefect.deployment.ready

events. If you see “not-ready” right when runs don’t start, that’s a lack of polling workers. - Check Automations that could pause deployments/pools/queues or cancel runs:

prefect automation ls

If runs start but never complete - Check worker/infrastructure logs where your workers run (these aren’t in Cloud): - Kubernetes:

kubectl logs <worker-pod>

- Docker:

docker logs <worker-container>

- Look for: - Image/code/storage auth issues (Pending → Crashed or stuck Pending) - OOM/evictions/signals (Running → no completion) - Network/API connectivity flaps (missed heartbeats) - Consider enabling runner heartbeat detection + an Automation to mark dead runs as Crashed and alert you. This helps catch “zombie” runs when infra dies silently. Things likely relevant to your setup (GCP workers) - If you’re on GKE, ensure the worker Deployment/Pod is steady (not on preemptible nodes without PodDisruptionBudgets). Brief restarts will create

not-ready

events and leave runs in Scheduled/Late. - Make sure the worker is polling the same named work queue your deployment targets (or omit

--work-queue

to poll all queues in the pool). - After switching to Cloud, confirm your deployments are registered in the Cloud workspace and point to Cloud work pools (names can match but the Cloud pool must exist and be polled). A focused repro/diagnostic you can run now 1) Pick a run that didn’t start. In the run page, note the state timeline (Scheduled → Late? Pending?). 2) Open that deployment’s Activity tab at that time. Do you see

prefect.deployment.not-ready

events? 3) Confirm the deployment’s pool/queue:

prefect deployment inspect <flow>/<deployment> | grep -E "work_pool_name|work_queue_name|paused|concurrency"

4) Confirm workers are polling the same pool/queue and heartbeating recently:

prefect worker ls
prefect work-pool inspect <pool-name> | grep -i status
prefect work-queue ls --pool <pool-name>

5) If the run did start but never completed, grab the worker/pod/container logs around that time. If you can share: - One example flow run ID that didn’t start or didn’t complete - The deployment name, work pool name, and (if used) work queue name - Whether you have any Automations configured that pause/resume deployments or pools - Whether your workers are running on GKE (and if so, any autoscaling/preemptible setup) …I can pinpoint the likely root cause and suggest the exact fix. Key takeaways - “Not-ready” events are triggered by missing worker polls and correlate directly with runs not leaving Scheduled/Late. - They affect execution indirectly: no workers → no pickup; Automations can also react to those events to pause/resume. - Most fixes come down to ensuring the correct pool/queue is polled continuously, removing unintended concurrency limits, and watching for infra/log errors in the worker environment.

Short answer: a work pool shows NOT_READY when Prefect doesn’t see any ONLINE workers heartbeating for that pool. That status is driven by worker heartbeats, not by queue polling or runs being created. So it’s possible for runs to still get picked up (e.g., by short‑lived/run‑once workers or workers that can poll queues but aren’t heartbeating), while the pool stays NOT_READY most of the time. Why this happens commonly - Workers are short‑lived or started with --run-once. The pool is NOT_READY between runs; it only flips to READY briefly when a worker is online. - Workers are polling queues but not sending heartbeats (misconfig, env var disabling heartbeat, version mismatch, or a bug). Queue(s) will look READY, but the pool stays NOT_READY. - You’re inspecting a different pool than the one your deployment actually uses. - Workers are polling a different queue in the pool than you expect; queue can be READY while the pool is NOT_READY if no worker is heartbeating. What to check right now - Do you have any ONLINE workers for that pool?

prefect worker ls

You should see workers for

gcp-workload-dev-us-central1-process

with recent heartbeats. If the list is empty or heartbeats are stale, the pool will show NOT_READY. - Confirm the deployment points to this pool (and the queue, if set):

prefect deployment inspect <flow>/<deployment> | grep -E "work_pool_name|work_queue_name|paused"

- Check queue status vs pool status:

prefect work-queue ls --pool gcp-workload-dev-us-central1-process

If queues are READY but the pool is NOT_READY, your workers are polling (updating last_polled) but not heartbeating. - Make sure workers are long‑running and heartbeating. From a worker host, start one and watch:

export PREFECT_LOGGING_LEVEL=DEBUG
prefect worker start --pool gcp-workload-dev-us-central1-process

Within ~30s the pool should flip to READY. If it doesn’t: - Check the worker logs for heartbeat errors. - Verify env vars aren’t disabling heartbeats (e.g., unusually large heartbeat intervals). - Confirm

prefect version

on the worker is 3.x and matches what your deployments expect. A couple of clarifying questions to pinpoint this: - How are you starting your workers (exact command/flags)? Are you using --run-once or an autoscaler that scales to zero? - When you run

prefect worker ls

, do you see ONLINE workers under

gcp-workload-dev-us-central1-process

with recent heartbeats? - Can you share the

prefect version

from the machine/container running your worker? - Do your deployments target a specific work queue? If so, is that the queue your workers are polling? If you run a single long‑lived worker with the command above and the pool still stays NOT_READY, that suggests a heartbeat issue (network, auth, or bug). In that case, grab the worker logs around startup so we can dig in.

Short answer: your workers are fine. That combo of “AwaitingConcurrencySlot” and “Aborted submission… Server returned a non-pending state 'SCHEDULED'” means the server is enforcing a concurrency limit and refusing to let the run move to Pending/Running. Everything is queued behind a limit somewhere. Why you see this - The server puts flow runs into a SCHEDULED state named “AwaitingConcurrencySlot” when there’s no available slot. - The worker tries to move the run to Pending; the server says “nope, still SCHEDULED,” so the worker logs “non‑pending state 'SCHEDULED'” and backs off. - This repeats until a slot opens. Most common culprits 1) Deployment-level concurrency limit set to 0 or too low (very common when moving configs). 2) Work pool concurrency limit set too low or 0. 3) Global or tag-based concurrency limits at capacity. 4) “Ghost” runs occupying slots (cancelled/crashed runs that will release their lease after the grace period, usually a few minutes). Quick triage checklist - Check the work pool your workers are polling:

prefect work-pool ls
prefect work-pool inspect <your-pool-name>
# If you see a concurrency_limit that’s too low or 0:
prefect work-pool set-concurrency-limit <your-pool-name> 10
# Or clear it entirely:
prefect work-pool clear-concurrency-limit <your-pool-name>

Docs: CLI: work-pool - Inspect a deployment that’s stuck and look for a concurrency_limit:

prefect deployment ls
prefect deployment inspect <flow-name>/<deployment-name> | grep -i concurrency

If it’s 0 or too small, raise or remove it. In Cloud UI: Deployments > pick the deployment > Concurrency. Docs: Deployments - Check global and tag-based concurrency limits:

prefect global-concurrency-limit ls
prefect concurrency-limit ls
prefect concurrency-limit inspect <tag>
# If a tag limit is wedged, you can reset/delete it:
prefect concurrency-limit reset <tag>
prefect concurrency-limit delete <tag>

Docs: - CLI: global-concurrency-limit - CLI: concurrency-limit - Look for runs currently in AwaitingConcurrencySlot to confirm the diagnosis:

prefect flow-run ls --filter "state_type:SCHEDULED state_name:AwaitingConcurrencySlot"

Notes and tips - If you just switched from self-hosted to Cloud and redeployed, double‑check you didn’t accidentally set

concurrency_limit: 0

on deployments or cap the work pool. - If there’s a big backlog and

concurrency_limit

is low (like 1), you’ll see lots of “AwaitingConcurrencySlot”—that’s expected until slots open. - If you suspect “ghost” slots, cancelling obviously stale runs will immediately free slots; otherwise they release automatically after the server’s lease grace period. - The “non‑pending state 'SCHEDULED'” log line is expected any time the server returns “AwaitingConcurrencySlot”; the worker will retry later. If you can share: - The output of

prefect work-pool inspect <pool>

- One

prefect deployment inspect <flow/deployment>

that’s stuck - The output of

prefect global-concurrency-limit ls

and

prefect concurrency-limit ls

…I can pinpoint exactly which limit is blocking you and suggest the fastest fix. References - States overview: States in Prefect - Deployments: Deployments - Work-pool CLI: work-pool CLI - Global concurrency limits CLI: global-concurrency-limit CLI - Tag-based concurrency limits CLI: concurrency-limit CLI

Thanks for the logs — that message is clear: Prefect Cloud is blocking new runs at the deployment level because the deployment’s concurrency limit has been reached. What this means - In Prefect 3.x, deployments can have a per-deployment concurrency limit. When the limit is full: - With collision strategy ENQUEUE: new runs wait and are rescheduled (“AwaitingConcurrencySlot”). - With collision strategy CANCEL_NEW: new runs are aborted immediately — which matches your “Aborted submission … Deployment concurrency limit reached” logs. Why your workers look idle - The server is telling the worker not to start the job because the deployment has no free concurrency slots (and likely uses CANCEL_NEW). The worker is healthy; the runs are being blocked upstream by the concurrency policy. How to fix it (pick one) 1) If you want runs to queue instead of aborting - Redeploy the deployment with ENQUEUE and an appropriate limit:

prefect deploy ./path/to/flow.py:my_flow \
  --name my-deployment \
  --concurrency-limit 5 \
  --collision-strategy ENQUEUE

2) If you want to raise or remove the limit - Redeploy with a higher limit:

prefect deploy ./path/to/flow.py:my_flow \
  --name my-deployment \
  --concurrency-limit 20

- Or remove it programmatically (set to null) without redeploying:

from uuid import UUID
from prefect.client.orchestration import get_client
from prefect.client.schemas.actions import DeploymentUpdate

async def remove_limit(deployment_id: str):
    async with get_client() as client:
        await client.update_deployment(
            deployment_id=UUID(deployment_id),
            deployment=DeploymentUpdate(concurrency_limit=None, concurrency_options=None),
        )

How to see what’s set right now - In Cloud UI: open the deployment’s page and check the Concurrency/Settings section for “concurrency limit” and “collision strategy.” - Via CLI:

prefect deployment ls
prefect deployment inspect "your-flow-name/your-deployment-name"

Look for

concurrency_limit

and

concurrency_options.collision_strategy

. Notes and gotchas - If you recently moved from self-hosted to Cloud, your deployment definitions may have carried over a conservative limit or CANCEL_NEW. Switching to ENQUEUE will keep runs from aborting and let them wait for a slot. - If, after fixing deployment concurrency, runs still queue/abort, also check global or tag-based concurrency limits that might be in play (though your message specifically points to a deployment limit). - Make sure your Prefect version is reasonably up to date (

prefect version

) so leasing/slot handling works as expected. Docs and references - Deployments (concepts, including concurrency): Deployments – Prefect 3.x - CLI (deploy): Deployment CLI reference - Concurrency CLI (global/tag): Concurrency limits CLI and Tag-based concurrency limits - Orchestration source (server behavior when limits are hit): - SecureFlowConcurrencySlots - ValidateDeploymentConcurrencyAtRunning If you can share: - The deployment name(s) seeing this - Your desired behavior (queue vs abort) - Your Prefect version (

prefect version

) …I can give you the exact command or API call to adjust the limit and collision strategy.

Thanks for the detail — that’s frustrating. A couple things can cause “workers online but idle,” and hitting API limits can make it look like they’re blocked. Here’s a focused plan to get you unstuck and reduce API pressure. Quick checks to confirm connectivity and configuration - Verify your Cloud API URL is the full account/workspace path (common source of “connected but idle”):

export PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/<ACCOUNT-ID>/workspaces/<WORKSPACE-ID>>"
  export PREFECT_API_KEY="<redacted>"

Common mistakes: missing

/api

, singular

account/workspace

instead of

accounts/workspaces

. Docs: Settings & profiles - Confirm workers are polling the same work pool (and queue) as your deployments:

prefect work-pool ls

Then check in the UI that your deployments target that pool/queue and have scheduled runs. Docs: Work pools • Deployments - Check the pool concurrency limit is > 0 and not fully consumed. If the pool limit is reached, workers will appear idle. - From a worker host, quickly test basic network to Cloud:

curl -I <https://api.prefect.cloud/api>

If this times out or fails with 4xx/5xx repeatedly, share the output. About the “API limits” and why workers look idle - Prefect Cloud enforces fair-use rate limits. When the API returns 429, the Prefect client automatically retries with backoff and honors Retry-After. After repeated failures, the worker’s service loop also backs off (exponentially). During these backoff windows, workers will look idle even though they’re just waiting to retry. - In logs, this shows up as 429s and messages like “Backing off due to consecutive errors” and “Another attempt will be made in Xs.” Immediate mitigations to reduce API pressure - Increase the polling interval (trade responsiveness for stability):

export PREFECT_WORKER_QUERY_SECONDS=30   # default is 10

- Optionally increase HTTP retry tolerance:

export PREFECT_CLIENT_MAX_RETRIES=10     # default is 5

- If you’re running many workers against the same pool, scale down the number of workers or shard across multiple work pools to spread polling load. - Restart the workers after changing env vars. Verifications and next steps - Start/verify your worker is attached to the correct pool:

prefect worker start -p <pool-name>

(Command verified.) - Double-check pool list:

prefect work-pool ls

(Command verified.) - If you migrated from self-hosted to Cloud, ensure you actually deployed to the Cloud workspace. Artifacts created in your old server won’t be visible to Cloud workers. In Prefect 3.x, use

prefect deploy

or programmatic

flow.from_source(...).deploy(pool="...", work_queue="...")

against the Cloud-connected environment. Docs: Deployments Helpful docs - Workers: Workers guide - Work pools: Work pools - Settings: Settings & profiles - Quickstart (Cloud): Quickstart Can you share: - The exact worker command(s) you’re running (or how they’re started on GCP) - The pool/queue your deployments target - A few recent worker log lines around the time things “stopped,” especially any 429/401 or “backing off” messages - Your current

PREFECT_API_URL

(you can redact IDs) and

PREFECT_WORKER_QUERY_SECONDS

value With that, I can pinpoint whether this is a rate-limit backoff, a pool/queue mismatch, or something environmental.

Thanks for sharing those warnings. They mean your flow processes and workers can’t establish a WebSocket connection to Prefect Cloud’s events endpoint. When this connection is blocked, runs can still execute, but event data (including task run state updates and some logs/automations signals) may be delayed or lost. What’s happening - The events client tries to open a WSS connection to: wss://api.prefect.cloud/api/accounts/<account_id>/workspaces/<workspace_id>/events/in - If the network blocks WebSockets (HTTP upgrade) on 443, or TLS/CA certs are missing, connection attempts fail and you’ll see that warning in both flow and worker logs. - There is no HTTP fallback for events; if WSS can’t connect after retries, events won’t be delivered. Common causes in GCP - Egress firewall/NAT rules block or inspect WebSocket traffic on port 443. - Corporate proxy that allows HTTPS but blocks the HTTP Upgrade to WebSocket. - Missing CA certificates in the container/base image (common with minimal or Alpine images). - TLS interception without the appropriate CA installed. - DNS resolution issues or system clock skew (TLS handshake failures). Actionable checks - Verify base connectivity from a worker/flow environment:

curl -sS <https://api.prefect.cloud/api/health>
  nslookup api.prefect.cloud

- Test a WebSocket handshake (use one of these): - With websocat (if you can install it):

websocat -v \
      -H "Authorization: Bearer <PREFECT_API_KEY>" \
      <wss://api.prefect.cloud/api/accounts/<account_id>/workspaces/<workspace_id>/events/in>

- Or a quick Python test (inside the same container/VM):

import asyncio, os, ssl
    import websockets

    async def main():
        url = "<wss://api.prefect.cloud/api/accounts/<account_id>/workspaces/<workspace_id>/events/in>"
        headers = {"Authorization": f"Bearer {os.environ['PREFECT_API_KEY']}"}
        ssl_ctx = ssl.create_default_context()
        async with websockets.connect(url, extra_headers=headers, ssl=ssl_ctx) as ws:
            print("Connected OK")

    asyncio.run(main())

If this fails, it’s almost certainly a network/proxy/TLS trust issue. - Ensure CA certificates are present in your image/VM: - Debian/Ubuntu:

apt-get update && apt-get install -y ca-certificates

- Alpine:

apk add --no-cache ca-certificates && update-ca-certificates

- Confirm egress rules allow TCP 443 to api.prefect.cloud and permit the HTTP Upgrade to WebSocket (some security devices allow HTTPS but block upgrade). - If you use a proxy, ensure it supports WebSocket (CONNECT/Upgrade) to api.prefect.cloud:443. Some HTTP(S) proxies block or strip Upgrade. - Check system clock and DNS:

timedatectl status
  nslookup api.prefect.cloud

- Enable debug logs to see the root exception:

export PREFECT_DEBUG_MODE=1

Re-run and look for the underlying cause (TLS, DNS, proxy, refused, etc.). Important notes - This WebSocket warning is separate from the “Deployment concurrency limit reached” aborts you saw earlier. Fixing concurrency settings will stop the aborts; fixing WSS connectivity will restore full event visibility/automations. - Allowlist by domain (api.prefect.cloud) rather than IP; Cloud IPs can change. Docs and references - Events overview: Events – Prefect 3.x - Network access and egress guidance: Configure network access - Cloud troubleshooting: Troubleshoot Prefect Cloud If you can share how your workers run in GCP (GKE, GCE, Cloud Run/Functions) and whether a corporate proxy or egress firewall is in place, I can tailor the next steps (e.g., specific GCP firewall/NAT/proxy config or base image changes) and provide a minimal connectivity test you can bake into your image.

Got it — if this worked before and nothing changed in your network, let’s triage what else could cause the WSS failures and the “idle” behavior: Key points - The events WebSocket is not required for scheduling. Even if WSS can’t connect, workers should still poll the REST API and run flows. If nothing is running, there’s likely a separate cause (earlier: deployment concurrency limit, or API rate-limit backoff). - When the events client can’t connect, you’ll see that warning; if it’s 401/403/429 under the hood, the client will retry then log the warning. Let’s surface the root cause. Quick, targeted checks 1) Enable debug to reveal the actual error (HTTP status/TLS/etc.)

export PREFECT_DEBUG_MODE=1
# restart the worker / rerun a flow and capture a few lines around the WSS warning

2) Verify API key is valid (auth failures present as WSS connect failures too) - Rotate a fresh API key in Cloud and update the worker/flow env just to rule out a 401/403. - Confirm your

PREFECT_API_URL

is the Cloud API endpoint (not the UI), which yours appears to be:

<https://api.prefect.cloud/api/accounts/<ACCOUNT>/workspaces/<WORKSPACE>>

3) If you scaled up recently, check GCP egress/NAT capacity - Large numbers of workers/containers behind a single NAT can hit ephemeral port exhaustion and cause intermittent new outbound connections (affecting WSS handshakes first). - If applicable: reduce worker count temporarily or shard across multiple NATs; confirm NAT utilization in GCP. 4) Reduce API pressure to get workers out of backoff (separate from WSS)

export PREFECT_WORKER_QUERY_SECONDS=30   # default 10
export PREFECT_CLIENT_MAX_RETRIES=10     # default 5
# restart workers

Then watch worker logs for any “429 … Backing off … Another attempt will be made in Xs”. 5) Minimal WSS connectivity test from the same host/image

import asyncio, os, ssl
import websockets

async def main():
    url = "<wss://api.prefect.cloud/api/accounts/<ACCOUNT>/workspaces/<WORKSPACE>/events/in>"
    headers = {"Authorization": f"Bearer {os.environ['PREFECT_API_KEY']}"}
    ssl_ctx = ssl.create_default_context()
    async with websockets.connect(url, extra_headers=headers, ssl=ssl_ctx) as ws:
        print("Connected OK")

asyncio.run(main())

- If this succeeds, the client should also be able to connect. - If it fails, the exception will tell us if it’s auth (401/403), TLS, or connection timeout. 6) Check Prefect Cloud status - Just to rule out a transient service/C-Edge issue: Prefect Cloud status Also, since earlier you were seeing “Deployment concurrency limit reached,” I’d recommend: - Switch collision strategy to ENQUEUE or raise/remove the deployment’s concurrency limit so new runs aren’t aborted. - Then address API rate limiting by increasing

PREFECT_WORKER_QUERY_SECONDS

and reducing parallel workers on the same pool. Helpful docs - Events: Events - Network access: Configure network access - Cloud troubleshooting: Troubleshoot Prefect Cloud If you can share: - Prefect version on workers/flows - Whether you rotated the API key and retried - A short debug snippet around the WSS warning (one attempt is enough) - Rough worker count and whether they’re behind a single GCP NAT …I can pinpoint whether this is an auth regression, rate-limit side effect, or an egress/NAT saturation pattern and get you a concrete fix. If the debug shows an issue on our side, I’ll escalate immediately.

Thanks for the clarification — that helps. Since this is only happening inside your flow run containers and started ~30 minutes ago, let’s zero in on what can change at flow runtime vs the worker: Key points - Flow run containers must have their own

PREFECT_API_URL

and

PREFECT_API_KEY

. The worker’s env does not automatically propagate to the job container unless you set it in the deployment’s infrastructure env. - The WSS error by itself should not block execution. If runs aren’t starting, there’s likely an additional issue (e.g., earlier “deployment concurrency limit reached”, auth failure, or API backoff). - A sudden break that affects only run containers often points to: an image change (missing CA certs), a rotated/expired API key not injected into the deployment env, or a transient service/edge incident. Fastest triage in the flow run container Please run these inside the same image/environment your deployment uses (or add a small diagnostic step at flow start) to isolate the root cause: 1) Print Prefect settings/env (confirm the container has Cloud creds)

from prefect.settings import PREFECT_API_URL, PREFECT_API_KEY, PREFECT_DEBUG_MODE
import prefect, os

print("Prefect version:", prefect.__version__)
print("API URL:", PREFECT_API_URL.value())
print("API KEY set:", bool(PREFECT_API_KEY.value()))
print("DEBUG:", PREFECT_DEBUG_MODE.value())

2) Test REST connectivity from the container (auth + TLS)

import httpx, os, ssl
url = "<https://api.prefect.cloud/api/health>"
r = httpx.get(url, timeout=10.0)
print("Health status:", r.status_code, r.text[:200])

3) Test WebSocket explicitly (auth errors/tls show up here)

import asyncio, os, ssl, websockets

async def main():
    url = "<wss://api.prefect.cloud/api/accounts/<ACCOUNT-ID>/workspaces/<WORKSPACE-ID>/events/in>"
    headers = {"Authorization": f"Bearer {os.environ['PREFECT_API_KEY']}"}
    async with websockets.connect(url, extra_headers=headers, ssl=ssl.create_default_context()) as ws:
        print("WSS connected OK")

asyncio.run(main())

What to look for - API key missing/invalid: REST or WS will return 401/403. - TLS/CA issue: SSL or handshake error (common if the image changed and lost

ca-certificates

). - DNS/egress: connection timeouts or refused. - If REST works but WS fails, it’s usually proxy/egress inspection or cert trust — but since this worked before, check for an image change or expired/rotated key first. Common fixes that match your symptoms - Re-inject Cloud creds into the deployment’s infrastructure environment: - In the Cloud UI: Deployment -> Infrastructure -> Environment, add -

PREFECT_API_URL=<https://api.prefect.cloud/api/accounts/<ACCOUNT-ID>/workspaces/<WORKSPACE-ID>>

-

PREFECT_API_KEY=<newly-rotated-key>

- Then re-run a deployment to ensure the job container has them. - If you rebuilt the image recently, ensure CA certs are present: - Debian/Ubuntu:

apt-get update && apt-get install -y ca-certificates

- Alpine:

apk add --no-cache ca-certificates && update-ca-certificates

- Rotate the API key in Prefect Cloud and update it in the deployment env (rules out an expiring/invalid token). - Enable debug to surface the actual error behind the WSS warning:

export PREFECT_DEBUG_MODE=1

Re-run a flow; the logs will show if it’s 401/403/timeout/TLS. Also revisit the earlier blocker - You were hitting “Deployment concurrency limit reached”. If that’s still set to CANCEL_NEW, new runs will be aborted even if WSS is healthy. Consider switching to ENQUEUE or increasing/removing the limit so runs don’t abort while we sort out WSS. Docs: Deployments Double-check Cloud status - Just to rule out a transient incident aligning with your 30-minute window: Prefect Cloud status If you can share: - A recent flow run ID that shows the WSS warning - Whether you recently rebuilt the flow image

- Whether the deployment’s Infrastructure -> Environment has the API URL/KEY set - The output of the small connectivity test above (or any debug snippet around the warning) …I can pinpoint whether this is an auth regression in run containers, an image/cert issue, or something on our side and get you a concrete fix fast.

Yes—there are a few ways to reduce API pressure when a deployment is backlogged on concurrency slots. What’s happening - With collision strategy ENQUEUE, the server reschedules blocked runs to “AwaitingConcurrencySlot” and sets the next attempt for “now + wait_seconds”. The worker continues to poll on its own interval for runs in the near future. If you have many workers or a short polling interval, this can create a lot of API traffic while slots are saturated. Practical levers you can use now - Slow down worker polling (biggest win to reduce API calls):

# Increase the poll interval (default is 10s)
  export PREFECT_WORKER_QUERY_SECONDS=30

  # Narrow the lookahead window (default is 10s)
  export PREFECT_WORKER_PREFETCH_SECONDS=5

Trade-off: slightly higher scheduling latency, much less API churn. - Reduce parallel workers on the same pool - Many workers attached to one pool all poll independently. Consolidating or sharding pools can reduce polling fan-out. - Prefer ENQUEUE over CANCEL_NEW for deployment-level concurrency - ENQUEUE prevents constant abort/reattempt loops. It already reschedules with a server-side delay so the same run won’t be retried immediately. - Consider pausing/reducing schedules when backlogged - If this deployment can burst-create runs, use Cloud UI/CLI to pause or slow the schedule during spikes so you don’t accumulate a large backlog that repeatedly rechecks for slots. Docs: Deployments About the server’s reschedule delay - The server controls how long a blocked run waits before retrying for a slot (default ~30s). This is governed by a server setting: -

server.deployments.concurrency_slot_wait_seconds

(env:

PREFECT_SERVER_DEPLOYMENTS_CONCURRENCY_SLOT_WAIT_SECONDS

) - Source: Core policy enqueue logic • Setting definition: Server deployment settings - In Prefect Cloud, this is managed by the service and isn’t user-configurable today. That’s why tuning the worker’s polling knobs is the best customer-side mitigation. Optional: reduce retries under heavy rate limiting - If you’re frequently seeing 429s and want the client to back off sooner, you can trim retries (at the cost of higher chance of transient failures surfacing):

export PREFECT_CLIENT_MAX_RETRIES=3   # default 5

Docs: Settings & profiles Recommended starting point - Set:

PREFECT_WORKER_QUERY_SECONDS=30
  PREFECT_WORKER_PREFETCH_SECONDS=5

- Ensure your deployment uses ENQUEUE for collision strategy. - Reduce the number of workers hitting the same pool if you have many. Helpful references - Workers: Workers guide - Deployments and concurrency limiting: Deployments – concurrency limiting If you share how many workers you have on the affected pool and your current `PREFECT_WORKER_QUERY_SECONDS`/`PREFECT_WORKER_PREFETCH_SECONDS`, I can suggest tuned values to balance latency vs API load for your traffic pattern.

Understood — if you’re still getting 429s, let’s aggressively dial down API pressure from workers during backlog and remove the biggest amplifiers. High‑impact changes you can make now - Slow worker polling more drastically

export PREFECT_WORKER_QUERY_SECONDS=60      # try 60–120s during congestion
  export PREFECT_WORKER_PREFETCH_SECONDS=5    # keep the lookahead tight
  export PREFECT_WORKER_HEARTBEAT_SECONDS=60  # reduce heartbeat traffic
  export PREFECT_WORKER_CANCELLATION_POLL_SECONDS=300  # if WS cancel isn’t used

Docs: Settings reference • Worker settings source: worker.py - Reduce concurrent pollers per pool - Run fewer workers against the same work pool, or shard into multiple pools (each with fewer workers). Many workers on one pool multiply poll traffic. - If you must keep many workers, split work by queues and pin each worker to a single queue:

prefect worker start -p <pool> -q <queue-a>
    prefect worker start -p <pool> -q <queue-b>

Workers filtered to one queue fetch less and overlap less. Guide: Work pools - Prefer ENQUEUE on deployment concurrency - ENQUEUE avoids constant abort/reattempt loops and uses a server-side wait. The server reschedules blocked runs about every 30s (Cloud-controlled). - Server-side wait reference: Enqueue reschedule logic - Note: In Cloud this “concurrency_slot_wait_seconds” is service-side and not tenant-configurable today, so tune worker polling instead. - Pause or throttle run creation during spikes - If a single deployment bursts many runs, pause its schedule until concurrency frees up, then resume:

prefect deployment schedule pause "<flow-name>/<deployment-name>"
    prefect deployment schedule resume "<flow-name>/<deployment-name>"

CLI verified. Docs: Deployments - Optional client tweaks (for stability under 429s)

export PREFECT_CLIENT_MAX_RETRIES=10        # tolerate more transient 429s
  export PREFECT_CLIENT_RETRY_JITTER_FACTOR=0 # deterministic backoff if desired
  export PREFECT_API_ENABLE_HTTP2=true        # better connection reuse

Settings: Settings reference Architecture tips that help a lot - Run one “backlog worker” on its own pool for this deployment with long polling (e.g., QUERY_SECONDS=60–120) and keep your other pools/workers responsive. - If you have many flows queued in the same near-future window, widen the spread: reduce schedule frequency or stagger triggers to avoid piling into the same reschedule time. What’s not available today - There isn’t a per-worker “rate limit” or request batching knob beyond the settings above. - The server’s reschedule delay for ENQUEUE is not tenant-configurable in Cloud. If you can share: - How many workers are attached to the affected pool - Current values of QUERY_SECONDS, PREFETCH_SECONDS, HEARTBEAT_SECONDS - Whether workers are pinned to queues or polling the whole pool …I’ll propose concrete values (e.g., “reduce from N workers to M, set QUERY_SECONDS to 90, PREFETCH_SECONDS to 5, HEARTBEAT_SECONDS to 60”) tailored to your traffic. If you continue to hit 429s after those changes, we can also coordinate with our team to review your account’s API traffic profile.

Short answer: yes. Make Kubernetes stop retrying the Job and let Prefect reschedule/cancel the flow instead. What to change 1) Set your Kubernetes work pool’s base job template to not retry - Ensure

spec.restartPolicy: "Never"

(this is the default in Prefect’s template) - Set

spec.backoffLimit: 0

so Kubernetes will not restart the Job after an eviction In Prefect Cloud: - Work Pools → your Kubernetes pool → Edit → Advanced → Base job template (JSON) - Add/confirm:

{
  "variables": {
    "backoff_limit": {
      "default": 0,
      "type": "integer",
      "minimum": 0
    }
  },
  "job_configuration": {
    "job_manifest": {
      "spec": {
        "backoffLimit": "{{ backoff_limit }}",
        "template": {
          "spec": {
            "restartPolicy": "Never"
          }
        }
      }
    }
  }
}

Why this works - With

backoffLimit: 0

, the Prefect Kubernetes worker configures the flow-run process to reschedule on SIGTERM from an eviction instead of letting Kubernetes keep retrying the Job. If

backoffLimit > 0

, Kubernetes will keep restarting pods and you’ll see the loop you described. Extra safeguards (optional but recommended) - Flow-level timeout to prevent zombie runs:

from prefect import flow

@flow(timeout_seconds=3600, retries=2, retry_delay_seconds=300)
def my_flow():
    ...

- Job/pod watch timeouts in the work pool to avoid waiting forever:

{
  "variables": {
    "pod_watch_timeout_seconds": {
      "default": 300,
      "type": "integer"
    },
    "job_watch_timeout_seconds": {
      "default": 3600,
      "type": "integer"
    }
  }
}

- Automation to auto-cancel long-running runs (e.g., cancel any run “Running” > N minutes). Docs: Automations and Creating automations Docs and references - Kubernetes work pools guide - Customize base job templates - Cancel a workflow if it runs too long If you’re still seeing restarts after this, please share: - A copy of your work pool’s base job template (Advanced JSON) - Any overrides you set in deployment

job_variables

- A snippet of the Job manifest that actually ran (from

kubectl get job -o yaml

) I’m happy to review and pinpoint what’s causing the restarts.

Got it — thanks for clarifying. What you’re describing sounds like repeated pod evictions causing Prefect to immediately reschedule the flow run over and over. That can happen if: - The cluster keeps evicting the pod (node pressure, missing resource requests/limits, ephemeral storage pressure), and - The flow is allowed to keep retrying (rescheduling) by Prefect. Two ways to stop the loop immediately Pick one depending on your desired behavior: A) Fail on eviction (no reschedule) - Explicitly disable the “reschedule on SIGTERM” behavior so evictions fail the run instead of rescheduling it. - Add this to your Kubernetes work pool base job template variables:

{
  "variables": {
    "env": {
      "PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR": "crash"
    },
    "backoff_limit": {
      "default": 0,
      "type": "integer",
      "minimum": 0
    }
  },
  "job_configuration": {
    "job_manifest": {
      "spec": {
        "backoffLimit": "{{ backoff_limit }}",
        "template": {
          "spec": {
            "restartPolicy": "Never"
          }
        }
      }
    }
  }
}

Any non-"reschedule" value (e.g., "crash") will stop Prefect from rescheduling on SIGTERM/eviction. The run will go to Crashed/Failed and stop. B) Keep rescheduling, but cap it - Set a small number of flow retries so Prefect stops after N reschedules:

from prefect import flow

@flow(retries=2, retry_delay_seconds=120, timeout_seconds=3600)
def my_flow():
    ...

This gives you at most 2 reschedules before the flow fails. Add an automation safety net (optional but recommended) - Auto-cancel runs stuck in a retry/reschedule loop (e.g., more than 5 retries in 30 minutes). See Automations: - Automations concepts - Creating automations Kubernetes-side fixes to reduce evictions - Add resource requests/limits in your base job template so the scheduler can place pods on nodes with enough capacity and avoid eviction:

{
  "job_configuration": {
    "job_manifest": {
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "name": "prefect-job",
                "resources": {
                  "requests": {
                    "cpu": "500m",
                    "memory": "1Gi"
                  },
                  "limits": {
                    "cpu": "1",
                    "memory": "2Gi"
                  }
                }
              }
            ]
          }
        }
      }
    }
  }
}

- If evictions are due to ephemeral storage, also set ephemeral-storage requests/limits and/or adjust node pool sizing. Useful timeouts (to avoid zombie runs) - In work pool variables:

{
  "variables": {
    "pod_watch_timeout_seconds": {
      "default": 300,
      "type": "integer"
    },
    "job_watch_timeout_seconds": {
      "default": 3600,
      "type": "integer"
    }
  }
}

Docs for reference - Kubernetes work pools guide - Customize base job templates - Cancel a workflow if it runs too long To help pinpoint the exact cause, could you share: - Prefect version inside your flow job image (

prefect version

from the pod logs) - A snippet of your work pool’s base job template (Advanced JSON) - Whether your flow has

retries

configured (and how many) - One example Job manifest from

kubectl get job <name> -o yaml

(just the

spec

section is fine) If you want the quickest stop-gap: set

PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR

to

"crash"

in the work pool’s variables. That will stop the endless reschedules immediately while we fix the eviction root cause.

Thanks — that symptom (18+ reruns with

retries=3

) makes sense if the restarts are coming from Kubernetes evictions with Prefect’s “reschedule on SIGTERM” behavior enabled. Those reschedules do not consume your flow’s retry budget; they put the run into AwaitingRetry directly, so you can see many more than 3 attempts if the pod keeps getting evicted. How to stop the loop Pick one of these approaches: 1) Stop rescheduling on eviction (fail instead) - Force evictions to fail/crash the run (no auto-reschedule), then you decide if/when to rerun. - In your Kubernetes work pool base job template, set:

{
  "variables": {
    "env": {
      "PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR": "crash"
    },
    "backoff_limit": {
      "default": 0,
      "type": "integer",
      "minimum": 0
    }
  },
  "job_configuration": {
    "job_manifest": {
      "spec": {
        "backoffLimit": "{{ backoff_limit }}",
        "template": {
          "spec": {
            "restartPolicy": "Never"
          }
        }
      }
    }
  }
}

This will stop the repeated reschedules immediately on eviction. 2) Shift the cap to Kubernetes - Set a small

backoffLimit

(e.g., 1–2) in the job manifest. Prefect will automatically disable the reschedule-on-SIGTERM behavior when

backoffLimit > 0

, so Kubernetes will retry the Job a limited number of times and then stop. - Keep

restartPolicy: Never

. 3) Add a safety net with Automations - Create an automation to cancel a run if it hits too many reschedules (e.g., after 5 occurrences of AwaitingRetry/Retrying within 30 minutes). Docs: Automations and Creating automations Optional but helpful - Add a flow-level timeout so runs can’t churn indefinitely:

from prefect import flow

@flow(timeout_seconds=3600, retries=3, retry_delay_seconds=120)
def my_flow():
    ...

- Reduce evictions by setting realistic resource requests/limits in your base job template (CPU, memory, and ephemeral-storage), or adjusting the node pool. References - Kubernetes work pools guide - Customize base job templates - Cancel a workflow if it runs too long If you want, I can pinpoint exactly why you hit 18 reschedules. Could you share: - Prefect version inside the flow job container (

prefect version

from pod logs) - The work pool’s base job template (Advanced JSON) - Whether you set

PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR

anywhere else - One example Job

spec

from

kubectl get job <name> -o yaml

- The

@flow(...)

decorator showing your

retries

config Quickest immediate fix while we debug: set

PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR

to

"crash"

in the work pool env. That will stop the continuous reschedules right away.