https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> what about this error in prefect serv...
# ask-marvin
e
@Marvin what about this error in prefect server (self-deployed with kubernetes, prefect version: 3.6.15, helm-release: 2026.1.30002458)
Copy code
12:33:19.847 | ERROR   | docket.worker - ↩ [ 10731ms] mark_deployments_ready(work_queue_ids=...){mark_deployments_ready:work_pool:a6837e39-a499-4df4-b63f-a9bb28bb384e}
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/docket/worker.py", line 878, in _execute
    result = await execution.function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/server/models/deployments.py", line 1126, in mark_deployments_ready
    await session.execute(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute
    result = await greenlet_spawn(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
    result = context.switch(value)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/bulk_persistence.py", line 1660, in orm_execute_statement
    return super().orm_execute_statement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
    result = conn.execute(
             ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
    return meth(
           ^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2366, in _handle_dbapi_exception
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 585, in execute
    self._adapt_connection.await_(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
    return current.parent.switch(awaitable)  # type: ignore[no-any-return,attr-defined] # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
    value = await result
            ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 563, in _prepare_and_execute
    self._handle_exception(error)
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 513, in _handle_exception
    self._adapt_connection._handle_exception(error)
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 799, in _handle_exception
    raise error
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 550, in _prepare_and_execute
    self._rows = deque(await prepared_stmt.fetch(*parameters))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 177, in fetch
    data = await self.__bind_execute(args, 0, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 268, in __bind_execute
    data, status, _ = await self.__do_execute(
                      ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 257, in __do_execute
    return await executor(protocol)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "asyncpg/protocol/protocol.pyx", line 205, in bind_execute
TimeoutError
m
✅ thought for 291.7 seconds
Thanks for the traceback — this is a database statement timeout coming from Prefect Server’s background task that marks deployments READY. What’s happening - The log is from Docket running mark_deployments_ready (a Prefect Server background task) which updates deployments to READY and emits events. - The TimeoutError is from asyncpg when executing the UPDATE/SELECT inside that function — the DB didn’t finish within the configured command timeout. - Common triggers: large batches of deployments tied to a work pool, DB lock waits/contention, a busy or undersized Postgres, or too-low timeouts. Relevant code/docs - Source (v3.6.15): mark_deployments_ready - Runs a SELECT of NOT_READY deployments (by deployment_ids/work_queue_ids), then an UPDATE setting status=READY, last_polled, and emits events. - Settings reference (all DB/server settings): Prefect 3.x Settings Reference Server DB settings that control this - Statement/connection timeouts (seconds) - 
PREFECT_SERVER_DATABASE_TIMEOUT
(default 10.0) → asyncpg command_timeout - 
PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT
(default 5.0) - Pooling (SQLAlchemy) - 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
(default 5) - 
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
(default 10) - 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT
(default 30.0) - Connect args (Postgres-specific, useful with PgBouncer) - 
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE
- 
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_PREPARED_STATEMENT_CACHE_SIZE
- If you’re using PgBouncer in transaction mode, set the two cache sizes to 0. Quick mitigations to try - Raise the statement timeout and right-size the pool. For a moderate/large deployment count, these are reasonable starting points:
Copy code
PREFECT_SERVER_DATABASE_TIMEOUT=30.0
PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT=10.0
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=10
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=60.0
- If using PgBouncer in transaction pooling:
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE=0
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_PREPARED_STATEMENT_CACHE_SIZE=0
- In Helm, add these as env vars on the API/Server pod (adapt to your chart’s values keys):
Copy code
api:
  extraEnv:
    - name: PREFECT_SERVER_DATABASE_TIMEOUT
      value: "30.0"
    - name: PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT
      value: "10.0"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
      value: "10"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
      value: "20"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT
      value: "60.0"
    # If using PgBouncer in transaction mode:
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE
      value: "0"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_PREPARED_STATEMENT_CACHE_SIZE
      value: "0"
Operational checks - Ensure Postgres isn’t enforcing a lower server-side 
statement_timeout
that cancels queries earlier than Prefect’s setting. - Watch for lock waits/contention:
Copy code
SELECT pid, state, wait_event_type, wait_event, query
FROM pg_stat_activity
WHERE state <> 'idle' AND datname = current_database();
- Vacuum/analyze the 
deployment
table if it’s large or frequently updated:
Copy code
VACUUM ANALYZE deployment;
- Turn on SQL echo to see statements and timings from Prefect:
Copy code
PREFECT_SERVER_DATABASE_ECHO=true
Why this isn’t catastrophic - The function has retry logic; brief timeouts during load often succeed on retry. If you see these errors continuously, it’s a sign your DB timeouts/pool sizing are too low or the DB is under pressure. A few questions to help tailor the fix
- Roughly how many deployments are in the affected work pool(s)? - Postgres version and whether PgBouncer is in use (and in which pooling mode)? - Any DB/server-side 
statement_timeout
or pgbouncer 
query_timeout
configured? - Current values for the env vars above (if any), and whether the error is intermittent or constant. If you share those details, I can recommend tighter settings for your environment.
e
@Vincent LEGENDRE upvote
6 Views
Open in Slack
PreviousNext
Powered by