<@ULVA73B9P> We're using Prefect Cloud and running...
# ask-community
w
@Marvin We're using Prefect Cloud and running into 503s during our deployment process. This was working very solidly for a while, but recently it's started failing. We create a large number of deployments during our CI/CD workflow, and initially we are succeeding and then start to fail with 503s, which seemingly points to some sort of throttling or overwhelming of the downstream endpoint. Is there any resources or references on where to see where the potential bottleneck or issue is?
m
thought for 290.4 seconds
Thanks for the detail — that pattern (lots of deployments created quickly, then intermittent 503s) does line up with hitting a throttle somewhere or a brief service strain. Here are the most useful places to look and concrete steps to de-bottleneck: Where to check - Prefect Cloud rate limits and plan quotas: Prefect Cloud rate limits. Creating a deployment is an API operation and large, parallel waves can exhaust per‑minute request budgets. - Prefect Cloud status and incidents: Prefect Cloud status. - Client retry behavior and knobs you can tune: Client retry settings. The SDK automatically retries 408/429/502/503 with exponential backoff and jitter, but you can increase patience when doing big CI/CD bursts. Quick mitigations (CI-friendly) - Stagger parallel deployment creation. If your CI spawns many
prefect deploy
in parallel, cap the concurrency or add short delays between batches. - Increase client retry budget in CI to ride out transient 503s/429s:
Copy code
export PREFECT_CLIENT_MAX_RETRIES=10
  export PREFECT_CLIENT_RETRY_JITTER_FACTOR=0.3
- If you’re running many small, nearly-identical deployments, consider consolidating where possible (e.g., parameterize a single deployment instead of N nearly-duplicate ones) to reduce API volume. Example: simple bash batching
Copy code
# Run up to 5 deployments at a time, with a short pause between waves
max_parallel=5
i=0
for spec in $(cat deployments.txt); do
  prefect deploy --name "$spec" &
  i=$((i+1))
  if [ "$i" -ge "$max_parallel" ]; then
    wait
    sleep 2
    i=0
  fi
done
wait
Helpful diagnostics - Turn up Prefect logging in CI to see retries and HTTP status codes:
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
This will show when the client is retrying 503s/429s and how often. - Confirm whether you’re seeing 429s as well as 503s — 429s definitively indicate rate limits; 503s can be transient service unavailability that the SDK will retry. Why this happens - Each deployment creation touches multiple API endpoints (flow registration, block docs, schedules, etc.). Firing a large number concurrently multiplies request volume and can hit account-level rate limits or briefly overwhelm the service, which explains “initially succeed, then fail” patterns as the minute bucket fills. References - Rate limits overview and plan-level numbers: Prefect Cloud rate limits - Client retry and tuning (env vars like
PREFECT_CLIENT_MAX_RETRIES
,
PREFECT_CLIENT_RETRY_JITTER_FACTOR
,
PREFECT_CLIENT_RETRY_EXTRA_CODES
): Client retry settings If you can share a bit more, I can tailor next steps: - Roughly how many deployments are created per run, and how many run in parallel? - Your Prefect Cloud plan/tier? - A snippet of the failing CI output (showing the 503s, and whether 429s appear too)? - Time window and workspace slug/org (feel free to DM) — we can check backend logs to confirm whether you’re hitting rate limits vs. something else. If this is blocking, we can also loop in support to review your workspace usage profile and recommend the best path (e.g., concurrency caps, batching, or plan adjustments).