https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I have around 300+ deployments in a ...
# marvin-ai
a
@Marvin I have around 300+ deployments in a Self hosted prefect environment on an Azure VM of 8 CPU core machine with Process work pool with 8 workers. 3 CPU cores are dedicated for Prefect Server and 5 CPU cores are dedicated for 1 work pool workers. Remaining 3 work pools workers are running on a separate VM. Postgres database running as a separate server. At a time, 30+ runs will get executed every 5 mins. 20+ runs are running smoothly, the remaining runs are getting started late (may be 1 min delayed) Sometimes - crashing with the below error only from the workers of the Linux VM Failed to submit flow run 'a7f20eb2-8c91-4419-a9a9-ab2a28686cb2' to infrastructure. Traceback (most recent call last): File "/data/Github/etl-data-engine/venv/lib/python3.12/site-packages/prefect/workers/base.py", line 1234, in _submit_run_and_capture_errors await self._give_worker_labels_to_flow_run(flow_run.id) File "/data/Github/etl-data-engine/venv/lib/python3.12/site-packages/prefect/workers/base.py", line 1517, in _give_worker_labels_to_flow_run await self._client.update_flow_run_labels(flow_run_id, labels) File "/data/Github/etl-data-engine/venv/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 931, in update_flow_run_labels response = await self.request( ^^^^^^^^^^^^^^^^^^^ File "/data/Github/etl-data-engine/venv/lib/python3.12/site-packages/prefect/client/orchestration/base.py", line 53, in request return await self._client.send(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/Github/etl-data-engine/venv/lib/python3.12/site-packages/prefect/client/base.py", line 361, in send response.raise_for_status() File "/data/Github/etl-data-engine/venv/lib/python3.12/site-packages/prefect/client/base.py", line 162, in raise_for_status raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.cause prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url Response: {'exception_message': 'Internal Server Error'} For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
m
✅ thought for 868.8 seconds
a
Still researching ?
m
Thanks for the detailed setup info — that helps a lot. Based on the symptoms, there are two likely contributors here: 1) runs starting “late” (~1 minute): this commonly comes from default scheduler/worker polling intervals and/or a thundering-herd of runs on the 5‑minute boundary 2) intermittent 500s on PATCH /flow_runs/{id}/labels (from Linux VM workers): this is almost always server/DB pressure — e.g., connection pool exhaustion or DB timeouts — not a client (worker) bug Below is a focused action plan you can try. I’ve split it into “fast wins” (no architecture changes) and “when you can” steps. Fast wins (no architecture changes) - Increase DB pool capacity and timeouts on the Prefect server - In Prefect 3.x, the server’s DB pool is tuned via env vars. The following are verified: - 
PREFECT_SERVER_DATABASE_CONNECTION_URL
- 
PREFECT_SERVER_DATABASE_TIMEOUT
(statement timeout, default 10s) - 
PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT
(default 5s) - 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
(default 5) - 
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
(default 10) - 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT
(default 30s) - 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE
(default 3600s) - Suggested starting point for your throughput:
Copy code
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
    export PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=40
    export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=60
    export PREFECT_SERVER_DATABASE_TIMEOUT=30
    export PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT=10
Note: 
PREFECT_API_DATABASE_*
aliases also work; stick to one prefix for clarity. - Reduce worker pickup latency (poll faster and prefetch) - These control how quickly workers grab new runs:
Copy code
export PREFECT_WORKER_QUERY_SECONDS=3          # default 10
    export PREFECT_WORKER_PREFETCH_SECONDS=30      # default 10
    export PREFECT_WORKER_HEARTBEAT_SECONDS=20     # default 30
- This will increase API traffic slightly, but will reduce “start late” behavior. - Lower the scheduler loop interval (if you’re running the built-in scheduler service) - The scheduler loop defaults to 60 seconds; at scale, that can manifest as ~1 minute delays when many runs are due at once. - Reduce it carefully (it increases DB load):
Copy code
export PREFECT_API_SERVICES_SCHEDULER_LOOP_SECONDS=10
If your environment prefers the SERVER prefix, 
PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS=10
is accepted too. - Trim event retention (keeps DB lean under write-heavy loads) - Events grow fast in busy installs and can stress vacuum/IO:
Copy code
export PREFECT_SERVER_EVENTS_RETENTION_PERIOD="2d"   # default 7d
- Revisit after a few days; if the DB calms down, you can lengthen slightly. - Monitor and confirm you’re not exhausting DB connections - On Postgres:
Copy code
SELECT count(*), state FROM pg_stat_activity GROUP BY state;
    SELECT pg_size_pretty(pg_total_relation_size('events'));
- If you increase the server pool sizes, ensure Postgres 
max_connections
has plenty of headroom. Why this helps your specific errors/delays - “Failed to submit flow run … _give_worker_labels_to_flow_run … 500 Internal Server Error”: that PATCH is small/fast. When it fails with 500, the server almost certainly hit a DB timeout or could not get a connection from the pool in time. Increasing pool_size/max_overflow/pool_timeout and DB statement/connection timeouts is the most effective mitigation. - “Some runs start ~1 minute late”: by default, the scheduler runs every 60s and workers poll every 10s. Under a 5‑minute burst (30+ due at once), this shows up as delays. Lowering the scheduler loop and worker query interval, plus prefetch, reduces these delays. Architecture and capacity tips (when you can) - Scale server capacity
- If the server VM is on 3 vCPU and handling 300 deployments plus busy services, consider giving it more CPU/RAM — or run a second API instance behind a load balancer. If you run multiple server processes/instances, only one should run “services” (scheduler, automations, etc.); others can be API-only. - Add more workers and/or split work pools - One Process work pool with 8 workers on 5 vCPU may be saturated launching 30+ runs every 5 minutes. Spreading runs across multiple pools or adding more worker capacity can reduce burst pressure. - Postgres hygiene - Ensure autovacuum is keeping up (especially on events, flow_run_state, task_run_state). If you see table bloat or long-running transactions, tune autovacuum and consider more aggressive VACUUM on hot tables. - If you use PgBouncer (optional), set prepared statement caches to 0 (transaction mode):
Copy code
export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE=0
    export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_PREPARED_STATEMENT_CACHE_SIZE=0
Useful reference and a related issue - A community report tying intermittent 500s to DB timeouts/pool pressure: Random 500 Internal Server Error in UI - Their workaround also involved increasing 
PREFECT_API_DATABASE_CONNECTION_TIMEOUT
and 
PREFECT_API_DATABASE_TIMEOUT
. A few clarifying questions to fine‑tune next steps - Which Prefect version are you on? If not on the latest 3.x, I recommend upgrading — we ship server/DB performance improvements regularly. - Rough DB size and Postgres version? Any PgBouncer in front? - What’s the CPU and IO utilization on the Postgres host when you see the 500s/delays? - Are all 30+ runs due exactly at the same minute (5‑minute boundary), or slightly staggered? If you can share a short window of server logs around a 500 failure (from the API VM), I can help pinpoint whether it’s a pool exhaustion vs. statement timeout vs. something else. Quick checklist to try first 1) Set the server DB pool/timeouts (block above), restart the server. 2) Lower worker query/prefetch intervals (block above), restart workers. 3) Optionally reduce scheduler loop to 10s, watch DB load. 4) Shorten events retention to 2d for now. 5) Watch pg_stat_activity and server logs during the 5‑minute burst. If the 500s persist after the pool/timeout changes, please share the server traceback around the 500; we’ll dig deeper.
a
@Marvin PREFECT_API_SERVICES_EVENT_PERSISTER_ENABLED='false' (from profile) PREFECT_API_SERVICES_TRIGGERS_ENABLED='false' (from profile) PREFECT_DEPLOYMENT_SCHEDULE_MAX_SCHEDULED_RUNS='1' (from profile) PREFECT_EVENTS_RETENTION_PERIOD='240000' (from profile) PREFECT_LOGGING_LEVEL='WARNING' (from profile) PREFECT_RESULTS_PERSIST_BY_DEFAULT='false' (from profile) PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT='10' (from profile) PREFECT_SERVER_DATABASE_TIMEOUT='30' (from profile) This is my config now. Can i try with this ?
m
✅ thought for 83.5 seconds
Short answer: you’re on the right track, but I’d add DB pool sizing and worker polling tweaks. Also, note one of your vars looks 2.x-era and will be ignored in Prefect 3. What you have now (good): - PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT=10 - PREFECT_SERVER_DATABASE_TIMEOUT=30 - PREFECT_EVENTS_RETENTION_PERIOD=240000 - PREFECT_LOGGING_LEVEL=WARNING - PREFECT_RESULTS_PERSIST_BY_DEFAULT=false - Disabling events/triggers services is fine if you don’t use event-driven features or automations. Add these to address the 500s and “late start” behavior: - Increase the server’s DB connection pool/overflow and wait time
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
  PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=40
  PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=60
These are the knobs that typically stop intermittent 500s on PATCH /flow_runs/{id}/labels under load. Code reference: ServerDatabaseSettings and SQLAlchemySettings - Reduce worker polling latency and prefetch a larger window
Copy code
PREFECT_WORKER_QUERY_SECONDS=3          # default 10
  PREFECT_WORKER_PREFETCH_SECONDS=30      # default 10
  PREFECT_WORKER_HEARTBEAT_SECONDS=20     # default 30
This helps workers pick up due runs faster when you have 30+ runs arriving every 5 minutes. - Optional: tighten the scheduler loop to reduce “minute-ish” delays
Copy code
PREFECT_API_SERVICES_SCHEDULER_LOOP_SECONDS=10
Note: lower loop_seconds increases DB load; pair this with the larger DB pool above. Important correction about this line: - PREFECT_DEPLOYMENT_SCHEDULE_MAX_SCHEDULED_RUNS='1' - That’s a Prefect 2.x-style setting and is ignored in Prefect 3.x. If your intent is to limit how many runs the scheduler schedules per loop, the Prefect 3 setting is 
PREFECT_SERVER_SERVICES_SCHEDULER_MAX_RUNS
(default 100). Generally, don’t set this to 1 — it can cause under-scheduling and perceived delays. A few trade-offs to be aware of: - PREFECT_API_SERVICES_EVENT_PERSISTER_ENABLED=false will stop persisting events; good for load, but event-driven automations and some reactive UI features won’t have historical events. If you later re-enable it, your 24h retention is sensible for high volume. - Keep using the SERVER_ prefix consistently (API_ aliases work, but consistency avoids confusion). After applying: - Restart the server and workers so settings take effect. - Watch Postgres during the 5‑minute bursts: - Connections vs max_connections - Query latency / timeouts - events, flow_run_state table growth If you still see 500s, please grab the server logs around a failing request so we can confirm if it’s connection pool exhaustion vs statement timeout. Related community report for similar 500s under load: Random 500 Internal Server Error in UI. Optional extra that helps at this scale: - Stagger schedules a bit (add small offsets/jitter) to avoid all 30+ runs landing on the same second of the 5‑minute boundary. Want me to propose exact env blocks for your two VMs (server and workers) based on this?
Open in Slack
PreviousNext
Powered by