Will Jackson
03/19/2026, 2:48 PMMarvin
03/19/2026, 2:49 PMMarvin
03/19/2026, 2:54 PMprefect deploy in parallel, cap the concurrency or add short delays between batches.
- Increase client retry budget in CI to ride out transient 503s/429s:
export PREFECT_CLIENT_MAX_RETRIES=10
export PREFECT_CLIENT_RETRY_JITTER_FACTOR=0.3
- If you’re running many small, nearly-identical deployments, consider consolidating where possible (e.g., parameterize a single deployment instead of N nearly-duplicate ones) to reduce API volume.
Example: simple bash batching
# Run up to 5 deployments at a time, with a short pause between waves
max_parallel=5
i=0
for spec in $(cat deployments.txt); do
prefect deploy --name "$spec" &
i=$((i+1))
if [ "$i" -ge "$max_parallel" ]; then
wait
sleep 2
i=0
fi
done
wait
Helpful diagnostics
- Turn up Prefect logging in CI to see retries and HTTP status codes:
export PREFECT_LOGGING_LEVEL=DEBUG
This will show when the client is retrying 503s/429s and how often.
- Confirm whether you’re seeing 429s as well as 503s — 429s definitively indicate rate limits; 503s can be transient service unavailability that the SDK will retry.
Why this happens
- Each deployment creation touches multiple API endpoints (flow registration, block docs, schedules, etc.). Firing a large number concurrently multiplies request volume and can hit account-level rate limits or briefly overwhelm the service, which explains “initially succeed, then fail” patterns as the minute bucket fills.
References
- Rate limits overview and plan-level numbers: Prefect Cloud rate limits
- Client retry and tuning (env vars like PREFECT_CLIENT_MAX_RETRIES, PREFECT_CLIENT_RETRY_JITTER_FACTOR, PREFECT_CLIENT_RETRY_EXTRA_CODES): Client retry settings
If you can share a bit more, I can tailor next steps:
- Roughly how many deployments are created per run, and how many run in parallel?
- Your Prefect Cloud plan/tier?
- A snippet of the failing CI output (showing the 503s, and whether 429s appear too)?
- Time window and workspace slug/org (feel free to DM) — we can check backend logs to confirm whether you’re hitting rate limits vs. something else.
If this is blocking, we can also loop in support to review your workspace usage profile and recommend the best path (e.g., concurrency caps, batching, or plan adjustments).