https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P>, We recently upgraded to Prefect vers...
# ask-marvin
r
@Marvin, We recently upgraded to Prefect version 3.6.24, and since then we’ve started encountering an issue. Some of our tasks are completing successfully (as confirmed by the logs), but their state in the Postgres database remains stuck as “running.” Because of this, our flows are not progressing to subsequent tasks. We did not face this issue prior to the upgrade. In addition to upgrading Prefect, we also migrated to a multi-worker architecture, where we are now running four workers along with a Redis server. Prefect server and services running separately. Could you help us identify the root cause of this behavior and suggest how we can resolve it?
m
❌ Error: GitHub authentication failed
Sorry, I encountered an error while processing your request. Please try again.
r
@Marvin try again, use your alternate tools
m
✅ thought for 11.8 seconds
Got it—I can dig in with the research and verification tools. Can you share what you want me to investigate or fix? Helpful details: - What you’re trying to do (e.g., deploy a flow, configure a worker, set concurrency) - Prefect version (I’ll assume Prefect 3.x unless you say otherwise) - Environment (local, Docker, Kubernetes, Cloud) - Error message or CLI command/output - Relevant code or 
prefect.yaml
I can: - Research a Prefect topic against the source code - Verify CLI commands and flags - Search GitHub issues/discussions for similar reports Tell me which path you want, and I’ll run the right tools.
r
@Marvin We recently upgraded to Prefect version 3.6.24, and since then we’ve started encountering an issue. Some of our tasks are completing successfully (as confirmed by the logs), but their state in the Postgres database remains stuck as “running.” Because of this, our flows are not progressing to subsequent tasks. We did not face this issue prior to the upgrade. In addition to upgrading Prefect, we also migrated to a multi-worker architecture, where we are now running four workers along with a Redis server. Prefect server and services running separately. Could you help us identify the root cause of this behavior and suggest how we can resolve it?
m
❌ Error: GitHub authentication failed
Sorry, I encountered an error while processing your request. Please try again.
r
@Marvin We recently upgraded to Prefect version 3.6.24, and since then we’ve started encountering an problem. Some of our tasks are completing successfully (as confirmed by the logs), but their state in the Postgres database remains stuck as “running.” Because of this, our flows are not progressing to subsequent tasks. We did not face this issue prior to the upgrade. In addition to upgrading Prefect, we also migrated to a multi-worker architecture, where we are now running four workers along with a Redis server. Prefect server and services running separately. Could you help us identify the root cause of this behavior and suggest how we can resolve it?
m
✅ thought for 276.1 seconds
Thanks for the detail — this symptom matches a known class of issues in Prefect 3.6.x where state update events don’t make it from workers into the database fast enough (or at all), especially after moving to multi-worker + Redis. You’ll see “task finished” in worker logs, but the server’s TaskRunRecorder hasn’t persisted the terminal state yet, so the DB still shows RUNNING and the flow can’t advance. Most likely causes in 3.6.24 - TaskRunRecorder contention/races in multi-worker setups: under concurrency, upserts can hit brief IntegrityErrors; without retry/batching, terminal states can be delayed or dropped. - Heartbeat starvation under CPU load: the background heartbeat/state-emission thread may not get scheduled promptly, so events reach the server late or not at all. - Redis event ordering/backlog: with multiple workers + Redis, the recorder service can lag or drop events when batching is too small, the queue grows, or services aren’t pointed at the same broker. - Causal ordering complexity in 3.6.24: can delay or reorder state updates in edge cases. This was simplified/removed in later 3.6.x. What I recommend 1) Quick stability win: upgrade patch-level to the latest 3.6.x - 3.6.27+ includes additional hardening around event ordering/recording. Since you’re on 3.6.24 already, this is a low-risk step and often resolves the “finished but RUNNING” symptom. 2) Verify the server services are running and healthy - Ensure your self-hosted server has these services up: TaskRunRecorder, EventPersister, Triggers, Scheduler, MarkLateRuns. - Confirm all services and all workers point to the same Redis (if using Redis as the message/event backend). - Make sure server and workers are on the same Prefect minor/patch (mismatches can surface as state sync issues). 3) Turn on DEBUG for a short reproduction window and check for telltale messages - On workers and on the server services:
Copy code
export PREFECT_LOG_LEVEL=DEBUG
# restart your workers and server services, then reproduce for ~10–20 minutes
- In server service logs (TaskRunRecorder), look for: - “IntegrityError” or “Retrying task_run upsert…” - “Dropping event … after … failed attempts” - Long gaps between “flushed N task runs” messages - In worker logs, look for: - “Failed to emit heartbeat” or unusually long delays between task finish and “state submitted” messages 4) Increase TaskRunRecorder/EventPersister throughput on the server - The defaults are conservative and can lag under concurrency. Scale batching on the server service(s):
Copy code
# TaskRunRecorder (no legacy aliases)
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=50
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=10
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1

# EventPersister (supports legacy aliases but prefer these)
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=100
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=10
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=2
- For very high concurrency (100+ concurrent tasks), consider:
Copy code
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=200
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=50
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=0.5
5) If CPU-bound workloads, relax heartbeat frequency a bit - This reduces scheduling pressure on the heartbeat/state-emission thread:
Copy code
export PREFECT_FLOWS_HEARTBEAT_FREQUENCY=60
- Note: Prefect enforces a 30s minimum; 60–120s is a good target for busy nodes. There is no separate “task heartbeat” setting in 3.x. 6) Check Postgres for stuck or delayed state persistence - Look for long-running RUNNING states and state timestamp anomalies: ``` -- Task runs stuck in RUNNING for >10 minutes SELECT id, flow_run_id, task_key, dynamic_key, state_type, state_name, state_timestamp, updated FROM task_run WHERE state_type = 'RUNNING' AND updated < NOW() - INTERVAL '10 minutes'
ORDER BY updated ASC LIMIT 100; -- Recent task run state creation lag SELECT NOW() - MAX(created) AS task_run_state_lag FROM task_run_state;
Copy code
- Check DB logs for deadlocks or unique constraint violations mentioning task_run/task_run_state.

7) Double-check version alignment and broker configuration
- Server services and workers should:
  - Run the same Prefect version (ideally latest 3.6.x).
  - Share the same Redis URL and credentials.
  - Have adequate DB connection pools on the server:
export PREFECT_SERVER_DATABASE_CONNECTION_POOL_MIN_SIZE=5 export PREFECT_SERVER_DATABASE_CONNECTION_POOL_MAX_SIZE=20 ``` Why this happens after your changes - The upgrade introduced more robust but also more active event recording; combined with your shift to multiple workers + Redis, you likely increased event volume and concurrency. With default batching (batch_size=1, flush_interval=5s), the recorder can fall behind. Under concurrency, brief DB conflicts can occur; without retries and larger batches, terminal states may be delayed long enough to stall downstream tasks. If you want, I can help triage live with a focused checklist. Helpful bits to share: - Confirm self-hosted vs Cloud, and exact Prefect version of: - workers - server API - server services image(s) - Whether workers are CPU-bound during these runs - Redis URL config locations for both workers and server services - A short log excerpt around a single “stuck” task run from: - The worker that ran it (DEBUG level) - The server TaskRunRecorder (DEBUG level) - Output of the SQL above for stuck RUNNING tasks Optional mitigations while you tune/upgrade - Temporarily lower concurrency or number of workers to reduce event pressure. - Restart the TaskRunRecorder service if you see it stuck with a large queue (after bumping batch sizes). - Upgrade to the latest 3.6.x first; if the problem persists, we can instrument the recorder and broker to pinpoint drops vs. delays. I’m happy to tailor exact env var manifests (Kubernetes, Docker Compose, systemd) if you share how you’re running the server services and workers.
r
@Marvin Your 4th solution seems right. Please review my default config and then confirm what to change
Copy code
PREFECT_API_MAX_FLOW_RUN_GRAPH_ARTIFACTS='10000' (from defaults)
PREFECT_API_MAX_FLOW_RUN_GRAPH_NODES='10000' (from defaults)
PREFECT_API_REQUEST_TIMEOUT='60.0' (from defaults)
PREFECT_API_SERVICES_CANCELLATION_CLEANUP_ENABLED='True' (from defaults)
PREFECT_API_SERVICES_CANCELLATION_CLEANUP_LOOP_SECONDS='20.0' (from defaults)
PREFECT_API_SERVICES_EVENT_LOGGER_ENABLED='False' (from defaults)
PREFECT_API_SERVICES_EVENT_PERSISTER_BATCH_SIZE='20' (from defaults)
PREFECT_API_SERVICES_EVENT_PERSISTER_ENABLED='True' (from defaults)
PREFECT_API_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL='5.0' (from defaults)
PREFECT_API_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE='1' (from defaults)
PREFECT_API_SERVICES_FOREMAN_DEPLOYMENT_LAST_POLLED_TIMEOUT_SECONDS='60' (from defaults)
PREFECT_API_SERVICES_FOREMAN_ENABLED='True' (from defaults)
PREFECT_API_SERVICES_FOREMAN_FALLBACK_HEARTBEAT_INTERVAL_SECONDS='30' (from defaults)
PREFECT_API_SERVICES_FOREMAN_INACTIVITY_HEARTBEAT_MULTIPLE='3' (from defaults)
PREFECT_API_SERVICES_FOREMAN_LOOP_SECONDS='15.0' (from defaults)
PREFECT_API_SERVICES_FOREMAN_WORK_QUEUE_LAST_POLLED_TIMEOUT_SECONDS='60' (from defaults)
PREFECT_API_SERVICES_LATE_RUNS_AFTER_SECONDS='0:00:15' (from defaults)
PREFECT_API_SERVICES_LATE_RUNS_ENABLED='True' (from defaults)
PREFECT_API_SERVICES_LATE_RUNS_LOOP_SECONDS='5.0' (from defaults)
PREFECT_API_SERVICES_PAUSE_EXPIRATIONS_ENABLED='True' (from defaults)
PREFECT_API_SERVICES_PAUSE_EXPIRATIONS_LOOP_SECONDS='5.0' (from defaults)
PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE='100' (from defaults)
PREFECT_API_SERVICES_SCHEDULER_ENABLED='True' (from defaults)
PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE='500' (from defaults)
PREFECT_API_SERVICES_SCHEDULER_LOOP_SECONDS='60.0' (from defaults)
PREFECT_API_SERVICES_SCHEDULER_MAX_RUNS='100' (from defaults)
PREFECT_API_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME='100 days, 0:00:00' (from defaults)
PREFECT_API_SERVICES_SCHEDULER_MIN_RUNS='3' (from defaults)
PREFECT_API_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME='1:00:00' (from defaults)
PREFECT_API_SERVICES_TASK_RUN_RECORDER_ENABLED='True' (from defaults)
PREFECT_API_SERVICES_TRIGGERS_ENABLED='True' (from defaults)
PREFECT_API_SSL_CERT_FILE='None' (from defaults)
PREFECT_API_TASK_CACHE_KEY_MAX_LENGTH='2000' (from defaults)
PREFECT_API_TLS_INSECURE_SKIP_VERIFY='False' (from defaults)
PREFECT_MESSAGING_BROKER='prefect_redis.messaging' (from profile)
PREFECT_MESSAGING_CACHE='prefect_redis.messaging' (from profile)
PREFECT_PROFILES_PATH='/home/infra/.prefect/profiles.toml' (from defaults)
PREFECT_RESULTS_DEFAULT_SERIALIZER='pickle' (from defaults)
PREFECT_RESULTS_PERSIST_BY_DEFAULT='true' (from profile)
PREFECT_RUNNER_CRASH_ON_CANCELLATION_FAILURE='False' (from defaults)
PREFECT_RUNNER_HEARTBEAT_FREQUENCY='180' (from defaults)
PREFECT_RUNNER_POLL_FREQUENCY='10' (from defaults)
PREFECT_RUNNER_PROCESS_LIMIT='5' (from defaults)
PREFECT_RUNNER_SERVER_ENABLE='False' (from defaults)
PREFECT_RUNNER_SERVER_HOST='localhost' (from defaults)
PREFECT_RUNNER_SERVER_LOG_LEVEL='ERROR' (from defaults)
PREFECT_RUNNER_SERVER_MISSED_POLLS_TOLERANCE='2' (from defaults)
PREFECT_RUNNER_SERVER_PORT='8080' (from defaults)
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='False' (from defaults)
PREFECT_SERVER_ANALYTICS_ENABLED='True' (from defaults)
PREFECT_SERVER_API_AUTH_STRING='********' (from defaults)
PREFECT_SERVER_API_BASE_PATH='None' (from defaults)
PREFECT_SERVER_API_HOST='0.0.0.0' (from profile)
PREFECT_SERVER_API_KEEPALIVE_TIMEOUT='5' (from defaults)
PREFECT_SERVER_API_PORT='4200' (from profile)
PREFECT_SERVER_CONCURRENCY_INITIAL_DEPLOYMENT_LEASE_DURATION='300.0' (from defaults)
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE='prefect_redis.lease_storage' (from profile)
PREFECT_SERVER_CONCURRENCY_MAXIMUM_CONCURRENCY_SLOT_WAIT_SECONDS='30.0' (from defaults)
PREFECT_SERVER_CORS_ALLOWED_HEADERS='*' (from defaults)
PREFECT_SERVER_CORS_ALLOWED_METHODS='*' (from defaults)
PREFECT_SERVER_CORS_ALLOWED_ORIGINS='*' (from defaults)
PREFECT_SERVER_CSRF_PROTECTION_ENABLED='False' (from defaults)
PREFECT_SERVER_CSRF_TOKEN_EXPIRATION='1:00:00' (from defaults)
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_APPLICATION_NAME='None' (from defaults)
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_PREPARED_STATEMENT_CACHE_SIZE='None' (from defaults)
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_SEARCH_PATH='None' (from defaults)
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE='None' (from defaults)
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CA_FILE='None' (from defaults)
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CERT_FILE='None' (from defaults)
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CHECK_HOSTNAME='True' (from defaults)
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED='False' (from defaults)
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_KEY_FILE='None' (from defaults)
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE='3600' (from defaults)
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT='30.0' (from defaults)
PREFECT_SERVER_DOCKET_NAME='prefect-server' (from defaults)
PREFECT_SERVER_DOCKET_URL='memory://' (from defaults)
PREFECT_SERVER_EPHEMERAL_STARTUP_TIMEOUT_SECONDS='20' (from defaults)
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING='prefect_redis.ordering' (from profile)
PREFECT_SERVER_EVENTS_MAXIMUM_EVENT_NAME_LENGTH='1024' (from defaults)
PREFECT_SERVER_LOGS_STREAM_OUT_ENABLED='False' (from defaults)
PREFECT_SERVER_LOGS_STREAM_PUBLISHING_ENABLED='False' (from defaults)
PREFECT_SERVER_SERVICES_DB_VACUUM_BATCH_SIZE='200' (from defaults)
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED='{'events'}' (from defaults)
PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS='3600.0' (from defaults)
PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD='90 days, 0:00:00' (from defaults)
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_MAX_FLUSH_RETRIES='5' (from defaults)
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE='50000' (from defaults)
PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED='True' (from defaults)
PREFECT_SERVER_SERVICES_REPOSSESSOR_LOOP_SECONDS='15.0' (from defaults)
PREFECT_SERVER_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS='5.0' (from defaults)
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE='1' (from defaults)
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL='5.0' (from defaults)
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE='1' (from defaults)
PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_HEARTBEAT_INTERVAL_SECONDS='5' (from defaults)
PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_RECONNECT_INTERVAL_SECONDS='10' (from defaults)
PREFECT_SERVER_SERVICES_TRIGGERS_READ_BATCH_SIZE='1' (from defaults)
PREFECT_SERVER_UI_SHOW_PROMOTIONAL_CONTENT='True' (from defaults)
PREFECT_SERVER_UI_V2_ENABLED='False' (from defaults)
PREFECT_SILENCE_API_URL_MISCONFIGURATION='False' (from defaults)
PREFECT_SQLALCHEMY_MAX_OVERFLOW='10' (from defaults)
PREFECT_SQLALCHEMY_POOL_SIZE='5' (from defaults)
PREFECT_TASKS_DEFAULT_NO_CACHE='False' (from defaults)
PREFECT_TASKS_DEFAULT_PERSIST_RESULT='None' (from defaults)
PREFECT_TASKS_DISABLE_CACHING='False' (from defaults)
PREFECT_TASKS_REFRESH_CACHE='False' (from defaults)
PREFECT_TASKS_RUNNER_PROCESS_POOL_MAX_WORKERS='None' (from defaults)
PREFECT_TASK_DEFAULT_RETRIES='0' (from defaults)
PREFECT_TASK_DEFAULT_RETRY_DELAY_SECONDS='0' (from defaults)
PREFECT_TASK_RUNNER_THREAD_POOL_MAX_WORKERS='None' (from defaults)
PREFECT_TASK_RUN_TAG_CONCURRENCY_SLOT_WAIT_SECONDS='10.0' (from defaults)
PREFECT_TASK_SCHEDULING_DEFAULT_STORAGE_BLOCK='None' (from defaults)
PREFECT_TASK_SCHEDULING_DELETE_FAILED_SUBMISSIONS='True' (from defaults)
PREFECT_TASK_SCHEDULING_MAX_RETRY_QUEUE_SIZE='100' (from defaults)
PREFECT_TASK_SCHEDULING_MAX_SCHEDULED_QUEUE_SIZE='1000' (from defaults)
PREFECT_TASK_SCHEDULING_PENDING_TASK_TIMEOUT='0:00:00' (from defaults)
PREFECT_TELEMETRY_ENABLE_RESOURCE_METRICS='True' (from defaults)
PREFECT_TELEMETRY_RESOURCE_METRICS_INTERVAL_SECONDS='10' (from defaults)
PREFECT_TEST_MODE='False' (from defaults)
PREFECT_TEST_SETTING='FOO' (from defaults)
PREFECT_UI_API_URL='<http://10.40.1.169:4200/api>' (from defaults)
PREFECT_UI_ENABLED='True' (from defaults)
PREFECT_UI_SERVE_BASE='/' (from defaults)
PREFECT_UI_STATIC_DIRECTORY='None' (from defaults)
PREFECT_UI_URL='<http://10.40.1.169:4200>' (from defaults)
PREFECT_UNIT_TEST_LOOP_DEBUG='True' (from defaults)
PREFECT_UNIT_TEST_MODE='False' (from defaults)
PREFECT_WORKER_CANCELLATION_POLL_SECONDS='120.0' (from defaults)
PREFECT_WORKER_DEBUG_MODE='False' (from defaults)
PREFECT_WORKER_ENABLE_CANCELLATION='False' (from defaults)
PREFECT_WORKER_HEARTBEAT_SECONDS='30.0' (from defaults)
PREFECT_WORKER_PREFETCH_SECONDS='10.0' (from defaults)
PREFECT_WORKER_QUERY_SECONDS='10.0' (from defaults)
PREFECT_WORKER_WEBSERVER_HOST='0.0.0.0' (from defaults)
PREFECT_WORKER_WEBSERVER_PORT='8080' (from defaults)
m
✅ thought for 180.1 seconds
Great—looking at your config, the two bottlenecks for state persistence are set to their most conservative defaults: - TaskRunRecorder (server-side) - PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE='1' - PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE='1' - PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL='5.0' - EventPersister (server-side) - You have the legacy aliases populated: - PREFECT_API_SERVICES_EVENT_PERSISTER_BATCH_SIZE='20' - PREFECT_API_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE='1' - PREFECT_API_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL='5.0' - These work, but I recommend also setting the canonical SERVER_ variants to keep things consistent going forward. Suggested changes (baseline for 4 workers with moderate concurrency) - Apply these on the server/services container(s), not on workers. Then restart the services.
Copy code
# TaskRunRecorder — increase throughput and reduce latency
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=50
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=10
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1

# EventPersister — align to higher throughput too
# Prefer canonical SERVER_ names; you can keep API_ as a fallback, but set these:
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=100
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=10
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=2

# Database pool — the server/services will open more concurrent DB connections under higher batching
# Prefer SERVER_ names over the deprecated top-level aliases you currently have set.
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
export PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=45
# (optional but common)
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=1800
If your actual peak concurrency is high (≈100+ concurrent tasks submitting state updates), use these instead:
Copy code
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=200
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=50
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=0.5

export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=250
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=50
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=1

export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=30
export PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=30
Notes on your current values - Keeping TRR at batch_size=1 and flush_interval=5s can easily create multi-second to multi-minute lag when many tasks finish around the same time. - Your EventPersister defaults (batch=20, read_batch=1, flush=5s) are okay but can backlog under load; bumping them helps the whole event pipeline. - You currently rely on deprecated top-level DB pool aliases (
PREFECT_SQLALCHEMY_POOL_SIZE
, 
PREFECT_SQLALCHEMY_MAX_OVERFLOW
). They still work, but I recommend switching to 
PREFECT_SERVER_DATABASE_SQLALCHEMY_*
for clarity and future-proofing. Operational tips - After applying, restart the Prefect server services and watch logs for: - Frequent “flushed N task runs” where N > 1, roughly every ~1s - Absence of IntegrityError/OperationalError around TaskRunRecorder - If workers are CPU-bound, consider also:
Copy code
export PREFECT_FLOWS_HEARTBEAT_FREQUENCY=60
This reduces scheduling pressure on the heartbeat thread but is optional for the throughput fix. If you can share your typical peak concurrency (total tasks that can complete per minute across the 4 workers), I can fine-tune these numbers further.
r
@Marvin I suspect, making these changes will increase DB connections and polling frequency. Will this have any impact on the DB connections or other things? its production server
m
✅ thought for 43.9 seconds
Short answer: yes, these changes can increase concurrent DB connections and the rate that the server flushes events—but in practice they usually reduce DB load and fix your symptom. Here’s what changes and how to roll it out safely in prod. What actually increases vs. decreases - TaskRunRecorder/EventPersister - READ_BATCH_SIZE: increases how many events the server reads from Redis per pull. This does not poll Postgres; it only affects broker consumption. - BATCH_SIZE: increases how many rows are written per DB transaction. This reduces the number of transactions per second vs. batch_size=1, which is typically easier on Postgres. - FLUSH_INTERVAL: with batching enabled, a shorter interval reduces end-to-end latency; it does not explode write volume. You’ll do fewer, larger commits instead of many tiny ones. - Database connection usage - Raising POOL_SIZE/MAX_OVERFLOW allows more concurrent DB sessions from the Prefect server/services process. If you set POOL_SIZE=20 and MAX_OVERFLOW=20, the worst case is up to ~40 connections from that process (multiply by the number of server/service pods or processes you run). Expected impact in production - Postgres - Fewer small commits and more batched writes typically reduce per-row overhead, lock churn, and WAL amplification compared to batch_size=1. - Higher pool size increases ceiling for concurrency. Make sure Postgres max_connections can accommodate: server pool (and any API workers) + other applications + admin connections. - If you run multiple server/service replicas, each replica has its own pool; multiply your pool sizing accordingly. - Redis - Higher READ_BATCH_SIZE increases stream/command throughput. Redis usually handles this easily; watch CPU if you’re near limits. - CPU/memory on server - Slightly higher CPU for batching/flush loops; small extra memory for in-flight batches; generally modest. Conservative rollout plan (prod-safe) 1) Phase 1: Fix latency with minimal DB risk - Only change batching; leave DB pool as-is (you have pool_size=5, max_overflow=10). - Apply on the server/services, then restart:
Copy code
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=50
   export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=10
   export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=1

   export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=100
   export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=10
   export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=2
- Observe for 15–30 minutes during load: - Postgres: active connections, lock waits, statement durations - Server logs: regular “flushed N task runs” with N > 1, reduced lag from task finish → state persisted 2) Phase 2: Only if you see “waited Xs for connection” or pool exhaustion - Increase pool conservatively:
Copy code
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=15
   export PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=15
   export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=45
- Recheck Postgres connection headroom before/after. 3) Phase 3: High concurrency (100+ tasks finishing per minute) - Consider:
Copy code
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_BATCH_SIZE=200
   export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_READ_BATCH_SIZE=50
   export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_FLUSH_INTERVAL=0.5

   export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20-30
   export PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20-30
- Tune up/down based on observed commit times and queue lag. How to monitor safely - Postgres (run on the DB) - Active connections by app (helps attribute load):
Copy code
SELECT application_name, count(*) 
  FROM pg_stat_activity 
  GROUP BY application_name ORDER BY 2 DESC;
- Long waits for connections/locks: ``` SELECT wait_event_type, wait_event, count(*) FROM pg_stat_activity WHERE state <> 'idle'
GROUP BY 1,2 ORDER BY 3 DESC;
Copy code
- Transaction volume trend:
SELECT date_trunc('minute', now()) AS ts, (SELECT SUM(xact_commit+xact_rollback) FROM pg_stat_database) AS tx_total; ``` - Prefect server logs (DEBUG temporarily) - Look for “flushed N task runs” frequency and any “IntegrityError”, “timeout waiting for connection”, or “dropping event”. Risk mitigations - Keep batch_size reasonable (50–200). Very large single commits can increase lock times. - Ensure Postgres max_connections > sum of: - Prefect server pool (and overflow) × number of server/service instances - Other application pools - 10–20 spare for admin/maintenance - If you’re near max_connections or in a multi-replica topology, consider PgBouncer. In that case, we can size the Prefect pool smaller and let PgBouncer smooth spikes. - Rollback is trivial: revert env vars and restart the server/services. Bottom line - The proposed batching changes primarily reduce transaction overhead while fixing your stuck RUNNING states; the net DB impact is usually equal or lower than before. - Only increase DB pool if you observe contention. Start with batching, observe, then adjust pool sizes incrementally. If you share: - Number of server/service instances (pods/containers) - Postgres max_connections - Typical peak of tasks completing per minute … I can give you a concrete ceiling for POOL_SIZE/MAX_OVERFLOW and confirm whether Phase 2 is needed.
5 Views
Open in Slack
PreviousNext
Powered by