https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I have installed prefect 3.7.3 versio...
# marvin-ai
f
@Marvin I have installed prefect 3.7.3 version on self-hosted. When I try to exectue a custom run and add a item to list, then it automatically start without possibilities to add the item to list. What do you recommend? Is the version 3.7.3 on beta?
m
✅ thought for 111.3 seconds
Prefect 
3.7.3
is not beta — it’s a normal/stable 3.x release. For your main issue: the run starting automatically before you can add items to a list is not the expected behavior. In Prefect 3.x, when you start a custom run from the UI, list parameters should let you click Add item first, then submit manually. It should not auto-start just because the list is empty. A few recommendations: 1. Check whether this is a UI/browser issue - Try in an incognito/private window - Try another browser - Hard refresh the page 2. Check your parameter definition If your flow parameter is a list, define it clearly with a default. For example:
Copy code
python
from prefect import flow
from pydantic import BaseModel, Field

class MyParams(BaseModel):
    items: list[str] = Field(default_factory=list)

@flow
def my_flow(items: list[str] | None = None):
    print(items)
Using 
default_factory=list
is often safer than leaving the list undefined. 3. Check whether you are running a deployment with parameter schema enforcement If 
enforce_parameter_schema
is enabled, Prefect validates inputs against the schema before submission. A malformed schema or required list field without a good default can cause odd form behavior. 4. Inspect the deployment If this is happening only for one deployment, inspect its parameters/schema. If you use CLI, the current command is:
Copy code
bash
prefect deployment inspect --help
Then inspect your deployment. Docs: - Form building / parameter forms - Run deployments manually Important note: I could not find evidence that this auto-start behavior is expected in 3.7.3, so this may be a bug or a deployment-specific schema problem. If you want, send me: - your flow parameter definition - whether you are using a deployment - a screenshot or exact steps in the UI and I can help narrow down whether it’s a schema problem or likely a UI bug.
f
@Marvin on image 3, when I try to add the item, clicking the text input then automatically launch the run.
m
✅ thought for 10.5 seconds
That definitely sounds like a UI bug, not expected behavior. Clicking into a text input for a list item should not launch the run. A few likely explanations: 1. The form is treating Enter/click as submit This can happen if the custom run dialog has a frontend bug in that Prefect UI version. 2. A bug specific to your self-hosted UI build Since you’re self-hosted, the API and UI version/build may matter. If the UI bundle is mismatched or cached, strange form-submit behavior can happen. 3. A schema/rendering bug for array parameters Especially if the parameter is: - a 
list[str]
- nested in a Pydantic model - required without a default - rendered as an array form in the custom run modal What I recommend: First, try a quick workaround - Define a default value for the list parameter, like 
[]
- Or pre-populate one item in the default - Or pass the list as raw JSON if that screen allows it Example:
Copy code
python
from prefect import flow
from pydantic import Field

@flow
def my_flow(items: list[str] = Field(default_factory=list)):
    print(items)
If using a model:
Copy code
python
from pydantic import BaseModel, Field

class Params(BaseModel):
    items: list[str] = Field(default_factory=list)
Second, verify whether it only happens for list fields Please test: - a simple 
str
parameter - an 
int
parameter - a 
list[str]
parameter If only 
list[...]
causes the run to start, that strongly points to a UI bug in the array input component. Third, check for version mismatch Make sure your self-hosted server/UI components are all on the same Prefect version (
3.7.3
), and clear browser cache. Fourth, consider upgrading Since this looks like a frontend issue, upgrading to the latest 3.x patch release is a good next step. You can also check the latest release notes: - Prefect releases Docs: - Form building - Run deployments If you want, send me the parameter definition for that flow, for example:
Copy code
python
@flow
def my_flow(items: list[str] = ...):
    ...
or your Pydantic model, and I can tell you whether the schema itself might be triggering this.
k
@Marvin What should my secret have so the backgroundservices can use my external elasticache cluster?
m
✅ thought for 69.9 seconds
If your background services need to use an external AWS ElastiCache Redis, the secret should usually contain the Redis connection info — either as a full URL or as password/token pieces, depending on how you deploy Prefect. The most important setting is: 
PREFECT_SERVER_DOCKET_URL
For ElastiCache, that typically looks like:
Copy code
text
<rediss://default:<AUTH_TOKEN>@<elasticache-endpoint>:6379/0>
If you're also using Redis for messaging/events, you may also need: 
PREFECT_REDIS_MESSAGING_URL
Example:
Copy code
text
<rediss://default:<AUTH_TOKEN>@<elasticache-endpoint>:6379/0>
Recommended secret contents If you are using Kubernetes/Helm, create a secret with at least one of these keys:
Copy code
yaml
apiVersion: v1
kind: Secret
metadata:
  name: prefect-redis
type: Opaque
stringData:
  docket-url: "<rediss://default:YOUR_AUTH_TOKEN@your-cache.xxxxxx.use1.cache.amazonaws.com:6379/0>"
  redis-password: "YOUR_AUTH_TOKEN"
  messaging-url: "<rediss://default:YOUR_AUTH_TOKEN@your-cache.xxxxxx.use1.cache.amazonaws.com:6379/0>"
Then wire those into Prefect values/env vars. Minimum needed for background services If your question is specifically about background services, the key one is:
Copy code
text
PREFECT_SERVER_DOCKET_URL
Without that, multiple background service processes cannot coordinate correctly. If using Helm/self-hosted values Typical values look like:
Copy code
yaml
redis:
  enabled: false

services:
  docket:
    existingSecret: prefect-redis
    existingSecretKey: docket-url

  broker: prefect_redis.messaging
  cache: prefect_redis.messaging

backgroundServices:
  runAsSeparateDeployment: true
If your chart supports messaging Redis config too, you may also set the messaging URL from the same secret. Important for ElastiCache - use 
rediss://
not 
redis://
if TLS is enabled - username is often 
default
- password is the Redis AUTH token - port is usually 
6379
If you want, I can help you build the exact 
values.yaml
snippet for your self-hosted Prefect installation if you paste your current Helm values.
k
@Marvin I am getting
Copy code
File "/usr/local/lib/python3.11/site-packages/prefect/server/api/background_workers.py", line 71, in background_worker                                                                                          │
│     await register_and_schedule_perpetual_services(                                                                                                                                                               │
│   File "/usr/local/lib/python3.11/site-packages/prefect/server/services/perpetual_services.py", line 163, in register_and_schedule_perpetual_services                                                             │
│     await docket.add(config.function, key=config.function.__name__)()                                                                                                                                             │
│   File "/usr/local/lib/python3.11/site-packages/docket/docket.py", line 387, in scheduler                                                                                                                         │
│     await execution.schedule(replace=False)                                                                                                                                                                       │
│   File "/usr/local/lib/python3.11/site-packages/docket/execution.py", line 520, in schedule                                                                                                                       │
│     reply = await schedule_script(                                                                                                                                                                                │
│             ^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                │
│   File "/usr/local/lib/python3.11/site-packages/redis/commands/core.py", line 6296, in __call__                                                                                                                   │
│     return await client.evalsha(self.sha, len(keys), *args)                                                                                                                                                       │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                       │
│   File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 781, in execute_command                                                                                                            │
│     result = await conn.retry.call_with_retry(                                                                                                                                                                    │
│              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                    │
│   File "/usr/local/lib/python3.11/site-packages/redis/asyncio/retry.py", line 69, in call_with_retry                                                                                                              │
│     return await do()                                                                                                                                                                                             │
│            ^^^^^^^^^^                                                                                                                                                                                             │
│   File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 722, in _send_command_parse_response                                                                                               │
│     return await self.parse_response(conn, command_name, **options)                                                                                                                                               │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                               │
│   File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 823, in parse_response                                                                                                             │
│     response = await connection.read_response()                                                                                                                                                                   │
│                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                   │
│   File "/usr/local/lib/python3.11/site-packages/redis/asyncio/connection.py", line 762, in read_response                                                                                                          │
│     raise response from None                                                                                                                                                                                      │
│ redis.exceptions.ClusterCrossSlotError: Keys in request don't hash to the same slot                                                                                                                               │
│ An exception occurred.
m
✅ thought for 132.1 seconds
That error means Prefect is connecting to a Redis Cluster-mode endpoint, and Redis is rejecting a multi-key operation:
Copy code
text
redis.exceptions.ClusterCrossSlotError: Keys in request don't hash to the same slot
Recommendation: use ElastiCache Redis with cluster mode disabled (standalone/replication mode), not cluster mode enabled, for Prefect background services. Why this happens: - Prefect background services use 
docket
and Redis-backed services - those perform Lua scripts / multi-key operations - Redis Cluster requires all keys in one operation to hash to the same slot - Prefect’s current Redis usage does not reliably satisfy that requirement for cluster-mode Redis So in practice: Works - ElastiCache Redis replication group with cluster mode disabled - a single Redis node - primary/replica Redis Does not work reliably - ElastiCache cluster mode enabled For self-hosted Prefect, configure: - 
PREFECT_SERVER_DOCKET_URL
- optionally 
PREFECT_REDIS_MESSAGING_URL
Example:
Copy code
text
PREFECT_SERVER_DOCKET_URL=<rediss://default:<AUTH_TOKEN>@<primary-endpoint>:6379/1>
PREFECT_REDIS_MESSAGING_URL=<rediss://default:<AUTH_TOKEN>@<primary-endpoint>:6379/0>
Notes: - use the primary endpoint of a non-cluster ElastiCache replication group - use 
rediss://
if TLS is enabled - keep Docket and messaging in separate DBs if you want, e.g. 
/1
and 
/0
So the fix is not changing the secret format — the secret can be fine — the issue is that the Redis topology is cluster-mode. If you want, I can help you translate this into: - a Kubernetes Secret - Helm 
values.yaml
- Docker Compose env vars Docs: - Prefect self-hosting - Docket
k
@Marvin can i enable all extra logger by default? instead of speciying python packages inside PREFECT_LOGGING_EXTRA_LOGGERS
m
✅ thought for 131.4 seconds
Short answer: no — Prefect does not support “enable all extra loggers” via 
PREFECT_LOGGING_EXTRA_LOGGERS
. 
PREFECT_LOGGING_EXTRA_LOGGERS
expects explicit logger names, for example:
Copy code
bash
PREFECT_LOGGING_EXTRA_LOGGERS=dask,scipy,numpy
It does not support: - 
*
- 
all
- wildcard patterns like 
mypackage.*
Those would be treated as literal logger names, not patterns. What I recommend: Option 1: add parent package loggers Because Python loggers are hierarchical, adding a parent logger often captures children too. Example:
Copy code
bash
PREFECT_LOGGING_EXTRA_LOGGERS=boto3,botocore,sqlalchemy,azure,google
That is usually enough for many libraries. Option 2: use a custom logging config If you want broader/default behavior, override Prefect’s logging config with a custom 
logging.yml
. Docs: - Logging customization A custom config can attach handlers more broadly than 
PREFECT_LOGGING_EXTRA_LOGGERS
. Option 3: rely on root logger for warnings/errors Unconfigured third-party loggers may still emit `WARNING`/`ERROR` through the root logger, but they will not automatically be treated like Prefect “extra loggers” for API log capture. So the practical answer is: - No built-in “all packages” switch - Best workaround: specify top-level parent loggers or use custom 
logging.yml
If you want, I can give you: 1. a ready-to-use 
logging.yml
that captures all library logs, or 2. a recommended 
PREFECT_LOGGING_EXTRA_LOGGERS
value for common AWS / Azure / db packages.
k
@Marvin
Copy code
File "asyncpg/protocol/protocol.pyx", line 205, in bind_execute                                                                                                                  │
│ 2026-06-19T11:59:17.632401867Z TimeoutError                                                                                                                                                                       │
│ 2026-06-19T11:59:28.294148380Z 11:59:28.290 | ERROR   | docket.dependencies - ↩ [ 10093ms] mark_deployments_ready(work_queue_ids=...){mark_deployments_ready:work_pool:cc909e3f-5706-4bc1-974b-a48eeb6e3371}      │
│ 2026-06-19T11:59:28.294177481Z Traceback (most recent call last):                                                                                                                                                 │
│ 2026-06-19T11:59:28.294180996Z   File "/usr/local/lib/python3.11/site-packages/docket/worker.py", line 1002, in _execute                                                                                          │
│ 2026-06-19T11:59:28.294183341Z     result = await execution.function(                                                                                                                                             │
│ 2026-06-19T11:59:28.294185293Z              ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                             │
│ 2026-06-19T11:59:28.294187715Z   File "/usr/local/lib/python3.11/site-packages/prefect/server/models/deployments.py", line 1317, in mark_deployments_ready                                                        │
│ 2026-06-19T11:59:28.294190295Z     result = await session.execute(select(locked.c.id, locked.c.status))                                                                                                           │
│ 2026-06-19T11:59:28.294192164Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                           │
│ 2026-06-19T11:59:28.294194721Z   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute                                                                           │
│ 2026-06-19T11:59:28.294196858Z     result = await greenlet_spawn(                                                                                                                                                 │
│ 2026-06-19T11:59:28.294198903Z              ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                 │
│ 2026-06-19T11:59:28.294201467Z   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn                                                                 │
│ 2026-06-19T11:59:28.294203613Z     result = context.switch(value)                                                                                                                                                 │
│ 2026-06-19T11:59:28.294205652Z              ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                 │
│ 2026-06-19T11:59:28.294207840Z   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2351, in execute                                                                                  │
│ 2026-06-19T11:59:28.294209957Z     return self._execute_internal(                                                                                                                                                 │
│ 2026-06-19T11:59:28.294211997Z            ^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                 │
│ 2026-06-19T11:59:28.294214318Z   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal                                                                        │
│ 2026-06-19T11:59:28.294216645Z     result: Result[Any] = compile_state_cls.orm_execute_statement(                                                                                                                 │
│ 2026-06-19T11:59:28.294219292Z                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                 │
│ 2026-06-19T11:59:28.294221977Z   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement                                                                     │
│ 2026-06-19T11:59:28.294224705Z     result = conn.execute(                                                                                                                                                         │
│ 2026-06-19T11:59:28.294227264Z              ^^^^^^^^^^^^^                                                                                                                                                         │
│ 2026-06-19T11:59:28.294229254Z   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1419, in execute                                                                                  │
│ 2026-06-19T11:59:28.294231245Z     return meth(                                                                                                                                                                   │
│ 2026-06-19T11:59:28.294233235Z            ^^^^^                                                                                                                                                                   │
│ 2026-06-19T11:59:28.294235181Z   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection                                                                   │
│ 2026-06-19T11:59:28.294237640Z     return connection._execute_clauseelement(                                                                                                                                      │
│ 2026-06-19T11:59:28.294239889Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                      │
│ 2026-06-19T11:59:28.294242801Z   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement                                                                   │
│ 2026-06-19T11:59:28.294245657Z     ret = self._execute_context(                                                                                                                                                   │
│ 2026-06-19T11:59:28.294247654Z           ^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                   │
│ 2026-06-19T11:59:28.294249755Z   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context                                                                         │
│ 2026-06-19T11:59:28.294251735Z     return self._exec_single_context(
m
✅ thought for 109.5 seconds
This error is different from the Redis one — this one points to Postgres query timeout in a background service. The important part is:
Copy code
text
asyncpg TimeoutError
...
mark_deployments_ready
...
session.execute(select(locked.c.id, locked.c.status))
What it means: - Prefect background services are trying to mark deployments as ready - that operation hits Postgres - the query is taking too long - 
asyncpg
times out before it finishes In Prefect 3.x, a likely cause is that the default DB statement timeout is too low for your workload. What I recommend first Increase:
Copy code
text
PREFECT_API_DATABASE_TIMEOUT
For example:
Copy code
bash
PREFECT_API_DATABASE_TIMEOUT=60
You may also want:
Copy code
bash
PREFECT_API_DATABASE_CONNECTION_TIMEOUT=10
If you are running in Kubernetes/Helm, add those env vars to the API/background services. Why this helps Prefect uses this timeout for Postgres commands. Background tasks like 
mark_deployments_ready
can take longer than the default, especially if: - you have many deployments - the database is under load - there is lock contention - the database is underprovisioned - table bloat/index/maintenance issues exist Also check your Postgres health I’d strongly recommend checking: - CPU / memory / IOPS on Postgres - active connections - slow queries - locks - autovacuum health Also consider increasing pool size If you have many background/API operations, these may help too:
Copy code
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
But only increase these if your Postgres instance can handle more connections. Most likely root causes 1. DB statement timeout too low 2. Postgres under heavy load 3. too many deployments/work queues 4. lock contention in the deployment tables 5. insufficient DB maintenance My suggested next step order 1. set 
PREFECT_API_DATABASE_TIMEOUT=60
2. restart Prefect services 3. watch whether the error stops 4. if not, inspect Postgres performance and locks Relevant docs: - Self-hosting Prefect - Database maintenance If you want, I can help you with: - a Helm 
values.yaml
snippet - Docker Compose env vars - a checklist of Postgres queries to diagnose locks / slow queries
k
@Marvin i increased the timeout and still getting TimeoutError │ │ 121411.410 | ERROR | docket.worker - ↩️ [ 30174ms] cancel_subflow_run(UUID('1d26a860-a199-4a17-9f91-fde8f980e4a1')){06a35327-07b1-76f7-8000-3a087f6b7dd3} │ │ Traceback (most recent call last): │ │ File "/usr/local/lib/python3.11/site-packages/docket/worker.py", line 1002, in _execute │ │ result = await execution.function( │ │ ^^^^^^^^^^^^^^^^^^^^^^^^^ │ │ File "/usr/local/lib/python3.11/site-packages/prefect/server/services/cancellation_cleanup.py", line 96, in cancel_subflow_run │ │ await models.flow_runs.set_flow_run_state( │ │ File "/usr/local/lib/python3.11/site-packages/prefect/server/models/flow_runs.py", line 620, in set_flow_run_state │ │ run = await models.flow_runs.read_flow_run( │ │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ │ File "/usr/local/lib/python3.11/site-packages/prefect/server/models/flow_runs.py", line 205, in read_flow_run │ │ result = await session.execute(select) │ │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ │ File "/usr/local/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute │ │ result = await greenlet_spawn( │ │ ^^^^^^^^^^^^^^^^^^^^^ │ │ File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn │ │ result = context.switch(value) │ │ ^^^^^^^^^^^^^^^^^^^^^ │ │ File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2351, in execute │ │ return self._execute_internal( │ │ ^^^^^^^^^^^^^^^^^^^^^^^ │ │ File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal │ │ result: Result[Any] = compile_state_cls.orm_execute_statement(
m
✅ thought for 21.4 seconds
If increasing 
PREFECT_API_DATABASE_TIMEOUT
did not help, then this is probably not just a timeout-setting problem. Your new trace shows even a relatively simple DB read is timing out:
Copy code
text
prefect.server.services.cancellation_cleanup.cancel_subflow_run
...
models.flow_runs.read_flow_run
...
await session.execute(select)
...
TimeoutError
That usually points to one of these: 1. Postgres is overloaded or stalled Even basic queries are too slow. 2. Lock contention A query is waiting on locks, not just running slowly. 3. Connection pool / connection acquisition issues Prefect may be waiting on DB connections or blocked sessions. 4. Network issue between Prefect and Postgres Latency, dropped packets, DNS, SSL negotiation, proxy issues, etc. At this point I’d recommend treating it as a database health / contention issue, not a Prefect config-only issue. What to check next in Postgres Run these on the Postgres instance: See active queries / waits
Copy code
sql
SELECT pid, usename, application_name, state, wait_event_type, wait_event, query_start, now() - query_start AS runtime, query
FROM pg_stat_activity
WHERE datname = current_database()
ORDER BY query_start;
Look for blocked queries
Copy code
sql
SELECT blocked.pid AS blocked_pid,
       blocked.query AS blocked_query,
       blocking.pid AS blocking_pid,
       blocking.query AS blocking_query
FROM pg_stat_activity blocked
JOIN pg_locks blocked_locks ON blocked.pid = blocked_locks.pid
JOIN pg_locks blocking_locks
  ON blocked_locks.locktype = blocking_locks.locktype
 AND blocked_locks.database IS NOT DISTINCT FROM blocking_locks.database
 AND blocked_locks.relation IS NOT DISTINCT FROM blocking_locks.relation
 AND blocked_locks.page IS NOT DISTINCT FROM blocking_locks.page
 AND blocked_locks.tuple IS NOT DISTINCT FROM blocking_locks.tuple
 AND blocked_locks.virtualxid IS NOT DISTINCT FROM blocking_locks.virtualxid
 AND blocked_locks.transactionid IS NOT DISTINCT FROM blocking_locks.transactionid
 AND blocked_locks.classid IS NOT DISTINCT FROM blocking_locks.classid
 AND blocked_locks.objid IS NOT DISTINCT FROM blocking_locks.objid
 AND blocked_locks.objsubid IS NOT DISTINCT FROM blocking_locks.objsubid
 AND blocked_locks.pid != blocking_locks.pid
JOIN pg_stat_activity blocking ON blocking.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;
See connection pressure
Copy code
sql
SELECT count(*) AS total_connections,
       count(*) FILTER (WHERE state = 'active') AS active,
       count(*) FILTER (WHERE state = 'idle') AS idle
FROM pg_stat_activity;
Things to verify in your deployment - Postgres CPU / memory / disk IOPS - whether you are using PgBouncer - whether autovacuum is healthy - whether the Prefect DB is very large / bloated - whether many background services are running concurrently - whether API and background services share the same constrained DB Prefect-side things to try 1. Reduce concurrency temporarily If you scaled background services high, reduce replicas and test. 2. Increase DB pool carefully Only if Postgres can support it:
Copy code
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
3. Confirm the timeout env var is applied to the running container Sometimes the env var was changed in config but not actually applied. 4. Check database URL / network path If Postgres is external: - DNS resolution - SSL config - proxy/load balancer - cross-AZ latency Important observation Your first error was in 
mark_deployments_ready
, now it is in 
cancel_subflow_run
. Since multiple unrelated background services are timing out on DB access, that strongly suggests a broader Postgres bottleneck rather than a bug in one specific Prefect service. If you want, send me: - how you deployed Prefect (Helm / docker-compose / ECS / etc.) - your Postgres type/version - whether Postgres is RDS / Aurora / self-managed - current values for: - 
PREFECT_API_DATABASE_TIMEOUT
- 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
- 
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
and I can help you narrow down whether this looks like lock contention, pool starvation, or DB saturation.
k
@Marvin it seems like a lot of the queries are blocked. my deployment has a separate background service and also server
m
✅ thought for 13.9 seconds
That’s a strong signal this is lock contention, and your architecture likely contributes to it. If you have: - a server/API deployment - a separate background services deployment then both can be hitting the same Postgres tables at the same time. That setup is valid, but if the DB is small or background work is heavy, blocked queries can cascade into 
TimeoutError
. What this likely means - queries are not necessarily “slow” by themselves - they are waiting on locks - then Prefect/asyncpg times out while waiting Since you’re seeing this across multiple background tasks, I’d focus on reducing contention rather than just raising timeouts further. Recommendations 1. Run only one background services replica If you have more than one replica for background services, reduce to 
1
first. Why: - many background tasks coordinate through DB + Redis - more replicas can increase contention if the DB is already stressed 2. Keep API/server replicas modest too If the API is scaled out aggressively, it can increase concurrent DB writes/reads and worsen lock waits. 3. Check for long-running transactions This is the biggest thing to inspect in Postgres. Long transactions hold locks and make everything pile up behind them. Useful query:
Copy code
sql
SELECT pid,
       usename,
       application_name,
       state,
       xact_start,
       now() - xact_start AS xact_age,
       wait_event_type,
       wait_event,
       query
FROM pg_stat_activity
WHERE xact_start IS NOT NULL
ORDER BY xact_start;
Look for: - very old transactions - sessions 
idle in transaction
- long-running updates/deletes 4. Identify blockers This is the most important query right now:
Copy code
sql
SELECT blocked.pid AS blocked_pid,
       blocked.usename AS blocked_user,
       blocking.pid AS blocking_pid,
       blocking.usename AS blocking_user,
       blocked.query AS blocked_query,
       blocking.query AS blocking_query,
       blocked.wait_event_type,
       blocked.wait_event
FROM pg_stat_activity blocked
JOIN pg_locks blocked_locks ON blocked.pid = blocked_locks.pid
JOIN pg_locks blocking_locks
  ON blocked_locks.locktype = blocking_locks.locktype
 AND blocked_locks.database IS NOT DISTINCT FROM blocking_locks.database
 AND blocked_locks.relation IS NOT DISTINCT FROM blocking_locks.relation
 AND blocked_locks.page IS NOT DISTINCT FROM blocking_locks.page
 AND blocked_locks.tuple IS NOT DISTINCT FROM blocking_locks.tuple
 AND blocked_locks.virtualxid IS NOT DISTINCT FROM blocking_locks.virtualxid
 AND blocked_locks.transactionid IS NOT DISTINCT FROM blocking_locks.transactionid
 AND blocked_locks.classid IS NOT DISTINCT FROM blocking_locks.classid
 AND blocked_locks.objid IS NOT DISTINCT FROM blocking_locks.objid
 AND blocked_locks.objsubid IS NOT DISTINCT FROM blocking_locks.objsubid
 AND blocked_locks.pid <> blocking_locks.pid
JOIN pg_stat_activity blocking ON blocking.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;
5. Make sure Redis/Docket is configured correctly Since you separated server and background services, make sure they coordinate through Redis correctly via:
Copy code
text
PREFECT_SERVER_DOCKET_URL
If that is missing or misconfigured, orchestration can behave badly. But based on your current symptoms, the DB blocking is the bigger problem. 6. Verify DB sizing If this is a small Postgres instance, contention gets worse faster. Check: - CPU saturation - memory pressure - IOPS/storage latency - max connections 7. Reduce Prefect activity temporarily As a test: - pause some automation-heavy workloads - reduce worker churn - reduce concurrent cancellations/retries/scheduling events If lock contention improves, that confirms DB pressure from orchestration load. Practical next move I would do these in order: 1. scale background services to 1 replica 2. collect blocking query output from Postgres 3. identify the blocking PID/query
4. see whether it’s a long transaction, maintenance gap, or overloaded DB 5. only then tune pool sizes / timeouts further Important If you want high availability with separate background services, that’s okay, but it requires: - healthy Postgres - correctly configured external Redis for Docket - enough DB capacity for orchestration traffic If you paste: - number of API replicas - number of background service replicas - Postgres instance size/type - the output of the blocker query I can help interpret which part is causing the contention.
k
@Marvin what RDS instance should i pick? and also redis
m
✅ thought for 9.9 seconds
For self-hosted Prefect, the safe recommendation is: Postgres (RDS) - use regular PostgreSQL RDS or Aurora PostgreSQL - start with a general purpose instance, not the tiniest one - for production, I would usually start around: - 
db.t4g.medium
or 
db.t4g.large
for small environments - 
db.r6g.large
if you expect heavier orchestration load or want more headroom If you already see blocked queries and timeouts, I would not choose a micro/small instance. Redis - use ElastiCache Redis / Valkey with cluster mode disabled - do not use Redis cluster mode enabled for Prefect background services - start with: - 
cache.t4g.small
or 
cache.t4g.medium
for small production - replication group preferred - use 
rediss://
if TLS is enabled My practical sizing guidance Small production / dev team - RDS: 
db.t4g.medium
- Redis: 
cache.t4g.small
Moderate production - RDS: 
db.t4g.large
or 
db.r6g.large
- Redis: 
cache.t4g.medium
If you already have DB blocking issues - prefer 
db.r6g.large
over burstable classes - burstable (
t*
) is fine for lighter steady workloads, but if orchestration is active and constant, memory-optimized/graviton instances are safer For Redis topology Choose: - replication group - cluster mode disabled Avoid: - cluster mode enabled Because Prefect can hit 
ClusterCrossSlotError
with cluster-mode Redis. What I’d recommend for you specifically Since you already reported: - blocked Postgres queries - background service timeouts - separate server + background service deployments I’d lean toward:
Copy code
text
RDS: db.r6g.large
Redis: cache.t4g.small or cache.t4g.medium
Redis mode: cluster mode disabled
That’s a safer baseline than going too small. Extra notes - provision decent storage/IOPS for Postgres - enable monitoring on RDS - watch connections, CPU, freeable memory, read/write latency - make sure autovacuum is healthy If you want, I can give you a right-sized recommendation based on: - number of flow runs per day - number of workers - number of API/background replicas - expected concurrent runs That would be more accurate than a generic instance suggestion.
k
@Marvin it looks like the rds is overwhelmed with a lot of reads. should the backgroundservices deployment point to a read ony instance?
m
✅ thought for 10.2 seconds
No — do not point Prefect background services to a read replica / read-only instance. Background services need read-write access to the database. They do things like: - update flow run state - mark deployments ready - manage cancellations - write events / orchestration metadata - acquire locks / run transactions A read replica will not work correctly for that. So both: - the Prefect API/server - the background services deployment should point to the same writable primary Postgres endpoint. Why you’re seeing lots of reads Even though background services also write, many orchestration operations begin with reads, row-locking reads, and transactional checks. Under load, that can look like lots of reads while still requiring the primary DB. What to do instead 1. Scale up the primary DB If the primary is overwhelmed, that is the right thing to change. 2. Reduce contention / concurrency - keep background services at 1 replica initially - avoid overscaling API replicas - reduce excessive orchestration churn if possible 3. Tune Postgres / RDS - choose a larger instance - ensure enough IOPS/storage throughput - monitor locks and long-running transactions 4. Keep Redis external and correct Use Redis/ElastiCache for Docket coordination, but Postgres still remains the source of truth for orchestration state. Short version - Primary/writable DB only for background services - No read replica for Prefect server components - if reads are overwhelming RDS, the solution is bigger/faster primary DB or less contention, not redirecting background services to read-only If you want, I can help you choose between: - bigger RDS instance - more IOPS - Aurora vs RDS Postgres - API/background replica counts based on your current workload.
k
@Marvin after upgrading the DB I now get
Copy code
Traceback (most recent call last):                                                                                              │
│ 2026-06-19T13:18:02.222964634Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/site-packages/prefect/server/utilities/postgres_listener.py", line 167, in get_pg_notify_conn │
│ ection                                                                                                                                                                                                            │
│ 2026-06-19T13:18:02.222966133Z prefect-server-background-services-bcc477d97-sqhzv     conn = await asyncpg.connect(dsn_string, **connect_args)                                                                    │
│ 2026-06-19T13:18:02.222967244Z prefect-server-background-services-bcc477d97-sqhzv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                    │
│ 2026-06-19T13:18:02.222968643Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/site-packages/asyncpg/connection.py", line 2443, in connect                                   │
│ 2026-06-19T13:18:02.222969744Z prefect-server-background-services-bcc477d97-sqhzv     return await connect_utils._connect(                                                                                        │
│ 2026-06-19T13:18:02.222971029Z prefect-server-background-services-bcc477d97-sqhzv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                        │
│ 2026-06-19T13:18:02.222972446Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 1249, in _connect                               │
│ 2026-06-19T13:18:02.222973936Z prefect-server-background-services-bcc477d97-sqhzv     raise last_error or exceptions.TargetServerAttributeNotMatched(                                                             │
│ 2026-06-19T13:18:02.222975018Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 1218, in _connect                               │
│ 2026-06-19T13:18:02.222976753Z prefect-server-background-services-bcc477d97-sqhzv     conn = await _connect_addr(                                                                                                 │
│ 2026-06-19T13:18:02.222977849Z prefect-server-background-services-bcc477d97-sqhzv            ^^^^^^^^^^^^^^^^^^^^                                                                                                 │
│ 2026-06-19T13:18:02.222979113Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 1054, in _connect_addr                          │
│ 2026-06-19T13:18:02.222990009Z prefect-server-background-services-bcc477d97-sqhzv     return await __connect_addr(params, True, *args)                                                                            │
│ 2026-06-19T13:18:02.222991911Z prefect-server-background-services-bcc477d97-sqhzv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                            │
│ 2026-06-19T13:18:02.222993598Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 1099, in __connect_addr                         │
│ 2026-06-19T13:18:02.222995237Z prefect-server-background-services-bcc477d97-sqhzv     tr, pr = await connector                                                                                                    │
│ 2026-06-19T13:18:02.222996800Z prefect-server-background-services-bcc477d97-sqhzv              ^^^^^^^^^^^^^^^                                                                                                    │
│ 2026-06-19T13:18:02.222998409Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 969, in _create_ssl_connection                  │
│ 2026-06-19T13:18:02.222999943Z prefect-server-background-services-bcc477d97-sqhzv     tr, pr = await loop.create_connection(                                                                                      │
│ 2026-06-19T13:18:02.223001332Z prefect-server-background-services-bcc477d97-sqhzv              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                      │
│ 2026-06-19T13:18:02.223003113Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1086, in create_connection                                      │
│ 2026-06-19T13:18:02.223004991Z prefect-server-background-services-bcc477d97-sqhzv     raise exceptions[0]                                                                                                         │
│ 2026-06-19T13:18:02.223006580Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1070, in create_connection                                      │
│ 2026-06-19T13:18:02.223008313Z prefect-server-background-services-bcc477d97-sqhzv     sock = await self._connect_sock(                                                                                            │
│ 2026-06-19T13:18:02.223010869Z prefect-server-background-services-bcc477d97-sqhzv            ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                            │
│ 2026-06-19T13:18:02.223012536Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 974, in _connect_sock                                           │
│ 2026-06-19T13:18:02.223013563Z prefect-server-background-services-bcc477d97-sqhzv     await self.sock_connect(sock, address)                                                                                      │
│ 2026-06-19T13:18:02.223014632Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 638, in sock_connect                                        │
│ 2026-06-19T13:18:02.223015727Z prefect-server-background-services-bcc477d97-sqhzv     return await fut                                                                                                            │
│ 2026-06-19T13:18:02.223016911Z prefect-server-background-services-bcc477d97-sqhzv            ^^^^^^^^^                                                                                                            │
│ 2026-06-19T13:18:02.223018147Z prefect-server-background-services-bcc477d97-sqhzv   File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 678, in _sock_connect_cb                                    │
│ 2026-06-19T13:18:02.223019348Z prefect-server-background-services-bcc477d97-sqhzv     raise OSError(err, f'Connect call failed {address}')                                                                        │
│ 2026-06-19T13:18:02.223020509Z prefect-server-background-services-bcc477d97-sqhzv ConnectionRefusedError: [Errno 111] Connect call failed ('172.19.49.135', 5432)
m
✅ thought for 12.9 seconds
This is a different problem now: Prefect is trying to open a dedicated PostgreSQL 
LISTEN/NOTIFY
connection and the TCP connection is being refused. The key line is:
Copy code
text
ConnectionRefusedError: [Errno 111] Connect call failed ('172.19.49.135', 5432)
That means: - DNS resolved to 
172.19.49.135
- Prefect reached that IP - but nothing accepted the connection on port 
5432
, or access is being rejected at the network layer This is not a query timeout anymore. Most likely causes after an RDS upgrade/change 1. The DB endpoint changed If you changed RDS instance / restored / promoted / replaced it, your Prefect connection string may still point to an old address. 2. Security group / network rules The new RDS instance may not allow inbound traffic from your Prefect pods/nodes. 3. Wrong endpoint type You may be pointing at: - an old instance endpoint - a reader endpoint - a temporary endpoint - a private address that is not reachable from the cluster 4. PgBouncer / proxy / service endpoint mismatch If you use a proxy, Prefect may now be pointing at a dead backend. What to check first 1. Verify the actual DB URL used by the background-services pod Check the value of your database connection env var in the running pod: - 
PREFECT_API_DATABASE_CONNECTION_URL
Make sure it points to the current writable endpoint. 2. Test connectivity from inside the background-services pod From the pod, test DNS and TCP reachability to the hostname in the DB URL. You want to confirm: - hostname resolves correctly - port 
5432
is reachable 3. Check RDS security groups Make sure the RDS security group allows inbound 
5432
from: - your EKS worker node security group, or - the subnet/CIDR your pods use, depending on your setup 4. Check the endpoint you should use For Prefect server and background services, use the writer/primary endpoint. Do not use: - a read-only endpoint - a stale instance endpoint from before the upgrade Why this appears in 
postgres_listener.py
 Prefect opens a dedicated Postgres connection for 
LISTEN/NOTIFY
in some background processes. That connection is separate from ordinary SQLAlchemy pooled queries, so if network or endpoint config is wrong, it can fail there even if other things previously worked intermittently. What I recommend - update all Prefect deployments to the correct new RDS writer endpoint - restart both: - server/API pods - background-services pods - verify security groups and subnet routing - make sure no old endpoint/IP is cached in env/config/secret If you want, paste your DB hostname format with sensitive parts redacted, for example:
Copy code
text
<http://myprefect-db.xxxxxx.us-east-1.rds.amazonaws.com|myprefect-db.xxxxxx.us-east-1.rds.amazonaws.com>
and tell me whether you are using: - RDS PostgreSQL - Aurora PostgreSQL - PgBouncer/proxy and I can tell you exactly which endpoint type Prefect should use.
k
@Marvin i am just getting File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 177, in fetch │ │ 2026-06-19T134026.869605632Z prefect-server-background-services-55bfc66954-bkpn9 data = await self.__bind_execute(args, 0, timeout) │ │ 2026-06-19T134026.869606920Z prefect-server-background-services-55bfc66954-bkpn9 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ │ 2026-06-19T134026.869608141Z prefect-server-background-services-55bfc66954-bkpn9 File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 268, in __bind_execute │ │ 2026-06-19T134026.869609432Z prefect-server-background-services-55bfc66954-bkpn9 data, status, _ = await self.__do_execute( │ │ 2026-06-19T134026.869610668Z prefect-server-background-services-55bfc66954-bkpn9 ^^^^^^^^^^^^^^^^^^^^^^^^ │ │ 2026-06-19T134026.869611881Z prefect-server-background-services-55bfc66954-bkpn9 File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 257, in __do_execute │ │ 2026-06-19T134026.869613120Z prefect-server-background-services-55bfc66954-bkpn9 return await executor(protocol) │ │ 2026-06-19T134026.869614470Z prefect-server-background-services-55bfc66954-bkpn9 ^^^^^^^^^^^^^^^^^^^^^^^^ │ │ 2026-06-19T134026.869615770Z prefect-server-background-services-55bfc66954-bkpn9 File "asyncpg/protocol/protocol.pyx", line 205, in bind_execute │ │ 2026-06-19T134026.869617104Z prefect-server-background-services-55bfc66954-bkpn9 TimeoutError │ │ 2026-06-19T134057.198163191Z prefect-server-background-services-55bfc66954-bkpn9 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ │ 2026-06-19T134057.198164505Z prefect-server-background-services-55bfc66954-bkpn9 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn │ │ 2026-06-19T134057.198165934Z prefect-server-background-services-55bfc66954-bkpn9 value = await result │ │ 2026-06-19T134057.198167165Z prefect-server-background-services-55bfc66954-bkpn9 ^^^^^^^^^^^^ │ │ 2026-06-19T134057.198168455Z prefect-server-background-services-55bfc66954-bkpn9 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 563, in _prepare_and_execute │ │ 2026-06-19T134057.198169868Z prefect-server-background-services-55bfc66954-bkpn9 self._handle_exception(error) │ │ 2026-06-19T134057.198171189Z prefect-server-background-services-55bfc66954-bkpn9 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 513, in _handle_exception │ │ 2026-06-19T134057.198172592Z prefect-server-background-services-55bfc66954-bkpn9 self._adapt_connection._handle_exception(error) │ │ 2026-06-19T134057.198185558Z prefect-server-background-services-55bfc66954-bkpn9 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 799, in _handle_exception │ │ 2026-06-19T134057.198188862Z prefect-server-background-services-55bfc66954-bkpn9 raise error │ │ 2026-06-19T134057.198190255Z prefect-server-background-services-55bfc66954-bkpn9 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 550, in _prepare_and_execute │ │ 2026-06-19T134057.198191621Z prefect-server-background-services-55bfc66954-bkpn9 self._rows = deque(await prepared_stmt.fetch(*parameters)) │ │ 2026-06-19T134057.198193236Z prefect-server-background-services-55bfc66954-bkpn9 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ │ 2026-06-19T134057.198195047Z prefect-server-background-services-55bfc66954-bkpn9 File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 177, in fetch │ │ 2026-06-19T134057.198196878Z prefect-server-background-services-55bfc66954-bkpn9 data = await self.__bind_execute(args, 0, timeout) │ │ 2026-06-19T134057.198199662Z prefect-server-background-services-55bfc66954-bkpn9 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ │ 2026-06-19T134057.198201545Z prefect-server-background-services-55bfc66954-bkpn9 File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 268, in __bind_execute │ │ 2026-06-19T134057.198203577Z prefect-server-background-services-55bfc66954-bkpn9 data, status, _ = await self.__do_execute( │ │ 2026-06-19T134057.198205454Z prefect-server-background-services-55bfc66954-bkpn9 ^^^^^^^^^^^^^^^^^^^^^^^^ │ │ 2026-06-19T134057.198207488Z prefect-server-background-services-55bfc66954-bkpn9 File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 257, in __do_execute │ │ 2026-06-19T134057.198209705Z prefect-server-background-services-55bfc66954-bkpn9 return await executor(protocol) │ │ 2026-06-19T134057.198210969Z prefect-server-background-services-55bfc66954-bkpn9 ^^^^^^^^^^^^^^^^^^^^^^^^ │ │ 2026-06-19T134057.198212406Z prefect-server-background-services-55bfc66954-bkpn9 File "asyncpg/protocol/protocol.pyx", line 205, in bind_execute │ │ 2026-06-19T134057.198213669Z prefect-server-background-services-55bfc66954-bkpn9 TimeoutError │ │ 2026-06-19T134057.202674754Z prefect-server-background-services-55bfc66954-bkpn9 134057.199 | ERROR | docket.worker - ↩️ [ 30112ms] cancel_subflow_run(UUID('77572350-f869-4133-a083-3486146ff552')){06a35 │ │ 468-e1f8-7986-8000-0ef593e85552}`
m
✅ thought for 13.3 seconds
Thanks — this brings us back to the same core issue: Postgres operations are still timing out under orchestration load. Because the traceback is still ending in:
Copy code
text
asyncpg/protocol/protocol.pyx ... bind_execute
TimeoutError
and it’s happening in background-service work like:
Copy code
text
cancel_subflow_run(...)
this strongly suggests: your database is still experiencing either - lock waits - heavy read/load pressure - connection pressure - or high-latency query execution —not a Redis problem, and not a Prefect “beta” issue. Important conclusion At this point, increasing timeout alone will not solve it reliably. It only makes Prefect wait longer for a DB that is already struggling. What I recommend now 1. Reduce orchestration pressure immediately For stabilization, set: - 1 API replica - 1 background services replica If you have more than that, reduce temporarily. Why: - multiple replicas can amplify DB contention - especially when cancellations, scheduling, deployment readiness, and state changes are all happening together 2. Check for blocked/long transactions in Postgres This is the most important next step. You already said queries looked blocked — that is likely the root cause. Run:
Copy code
sql
SELECT pid,
       usename,
       application_name,
       state,
       xact_start,
       now() - xact_start AS xact_age,
       wait_event_type,
       wait_event,
       query
FROM pg_stat_activity
WHERE datname = current_database()
ORDER BY xact_start NULLS LAST;
and:
Copy code
sql
SELECT blocked.pid AS blocked_pid,
       blocking.pid AS blocking_pid,
       blocked.state AS blocked_state,
       blocking.state AS blocking_state,
       blocked.wait_event_type,
       blocked.wait_event,
       blocked.query AS blocked_query,
       blocking.query AS blocking_query
FROM pg_stat_activity blocked
JOIN pg_locks blocked_locks ON blocked.pid = blocked_locks.pid
JOIN pg_locks blocking_locks
  ON blocked_locks.locktype = blocking_locks.locktype
 AND blocked_locks.database IS NOT DISTINCT FROM blocking_locks.database
 AND blocked_locks.relation IS NOT DISTINCT FROM blocking_locks.relation
 AND blocked_locks.page IS NOT DISTINCT FROM blocking_locks.page
 AND blocked_locks.tuple IS NOT DISTINCT FROM blocking_locks.tuple
 AND blocked_locks.virtualxid IS NOT DISTINCT FROM blocking_locks.virtualxid
 AND blocked_locks.transactionid IS NOT DISTINCT FROM blocking_locks.transactionid
 AND blocked_locks.classid IS NOT DISTINCT FROM blocking_locks.classid
 AND blocked_locks.objid IS NOT DISTINCT FROM blocking_locks.objid
 AND blocked_locks.objsubid IS NOT DISTINCT FROM blocking_locks.objsubid
 AND blocked_locks.pid <> blocking_locks.pid
JOIN pg_stat_activity blocking ON blocking.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;
3. Inspect RDS metrics Look specifically at: - CPU utilization - freeable memory - read latency - write latency - disk queue depth / IOPS - database connections If reads are overwhelming the DB, that confirms the bottleneck is still infra-side. 4. Make sure you are not overdriving Prefect If many flows are rapidly: - starting - cancelling - retrying - spawning subflows - updating state frequently then orchestration traffic can become very DB-heavy. Your log specifically mentions 
cancel_subflow_run
, so a workload with many subflow cancellations can create extra state churn. 5. Verify your DB class is truly adequate If you upgraded only slightly, it may still be undersized. For this kind of orchestration load, I’d lean toward: - 
db.r6g.large
minimum for moderate production - possibly larger if there is sustained activity 6. Consider Aurora PostgreSQL if load is consistently high Not required, but if you have lots of orchestration events and sustained activity, Aurora can provide better scaling characteristics than a very small RDS PostgreSQL instance. 7. Keep background services on the writer DB Just to reinforce: do not move them to a read replica.
My strongest practical recommendation right now To stabilize quickly: - scale API to 
1
- scale background services to 
1
- use a writable RDS endpoint only - inspect blocker queries in Postgres - check RDS read latency / CPU / connections - review whether your workload is generating excessive state transitions/cancellations/subflows If you want, I can help you create a triage checklist for: 1. Kubernetes replica settings 2. exact RDS metrics to inspect 3. SQL queries to identify the blocker 4. whether your workload pattern is too orchestration-heavy for current sizing
k
@Marvin i did try to do the cancel flow run before and the pod was deleted but the sattus never updated, would that be the issue?
m
✅ thought for 9.8 seconds
Yes — very possibly. If you previously tried to cancel a flow run, the execution pod got deleted, but the flow run state did not update, that can absolutely contribute to what you're seeing now. Why: - Prefect may keep trying to reconcile/cancel that run - background services can repeatedly revisit stuck flow/subflow state - cancellation cleanup tasks like 
cancel_subflow_run(...)
may keep firing - that adds more DB reads/writes and can worsen contention So this can be both: - a symptom of the DB/orchestration issue - and also a source of repeated background work that keeps pressure on the DB What likely happened The infrastructure disappeared, but Prefect never completed the corresponding state transition in the database. So Prefect still believes cleanup/reconciliation is needed. What I would check Find flow runs that are stuck in: - 
Cancelling
- 
Running
even though the pod is gone - subflows related to that cancelled parent run If you find many stuck runs, that would explain repeated cleanup attempts. Recommendation Look in the UI for: - flow runs stuck in cancelling/running - subflows of those runs - recent failed cancellations If it’s only a small number of stuck runs, cleaning them up can help reduce repeated background-service work. Important caution Do not delete DB rows manually unless you really know the schema implications. The safest approach is to update the run state through Prefect’s API/UI if possible. If the UI/API can still access the run, try marking it to a terminal state rather than leaving it stuck in cancellation-related states. Why this matches your logs You’re seeing errors around:
Copy code
text
cancel_subflow_run(...)
So yes — stale cancellation work for runs whose pods are already gone is very consistent with your symptoms. Short answer Yes, a previously cancelled run whose pod was deleted but whose Prefect state never updated could definitely be part of the problem. If you want, I can help you figure out the safest way to identify and clean up stuck cancelling/running flow runs in self-hosted Prefect.
k
@Marvin yes
m
✅ thought for 10.3 seconds
Yes — here’s the safest way to identify and clean up stuck flow runs in self-hosted Prefect. Goal Find runs that are still 
Running
or 
Cancelling
even though their infrastructure is gone, then move them to a terminal state. Recommended order 1. Use the Prefect UI first Look for runs in: - 
Cancelling
- 
Running
- possibly 
Pending
for a long time Check whether: - the worker pod/job no longer exists - the run has had no logs/heartbeats for a long time If a run is clearly dead, it is a candidate for manual cleanup. 2. Use the API/UI to change state if possible Safest approach is through Prefect’s normal state APIs, not direct DB edits. If you can access the run in the UI, try: - cancel again - mark as failed/crashed if that action is available in your setup 3. Find stuck runs directly in Postgres If you need to inspect the database, start read-only. A useful query is:
Copy code
sql
SELECT id,
       name,
       state_type,
       state_name,
       expected_start_time,
       start_time,
       end_time,
       created,
       updated
FROM flow_run
WHERE state_type IN ('RUNNING', 'CANCELLING')
ORDER BY updated ASC;
That shows the oldest stale runs first. You can also look for task runs if needed:
Copy code
sql
SELECT id,
       flow_run_id,
       name,
       state_type,
       state_name,
       created,
       updated
FROM task_run
WHERE state_type IN ('RUNNING', 'CANCELLING')
ORDER BY updated ASC;
4. Correlate with missing infrastructure For each stuck run, verify: - no Kubernetes job/pod still exists - no worker is still actively processing it If infra is definitely gone, that strengthens the case that the run is orphaned. 5. If API/UI cannot recover it, use great caution I do not recommend manual DB edits as the first choice. But if the system is stuck and API/UI cannot resolve the state, the next best step is to: - take a DB backup/snapshot first - update only a small number of obviously orphaned runs - move them to a terminal state consistently Because direct DB changes can be risky, I want to be careful here: I should not invent exact SQL state-transition updates without verifying your schema/state tables first. So before I suggest any SQL updates, I’d want one of these from you: - your Prefect version confirmation (
3.7.3
still?) - whether you’re comfortable using the REST API - or permission to help you inspect the schema/state representation first Best immediate diagnostic query This one is very useful for orphaned runs:
Copy code
sql
SELECT id,
       name,
       state_type,
       state_name,
       deployment_id,
       work_queue_id,
       created,
       updated,
       start_time,
       end_time
FROM flow_run
WHERE state_type IN ('RUNNING', 'CANCELLING')
  AND updated < now() - interval '30 minutes'
ORDER BY updated ASC;
That highlights stale runs that likely need intervention. My recommendation Start by: 1. listing stale 
RUNNING
/ 
CANCELLING
runs 2. checking whether their pods/jobs still exist 3. trying UI/API-based cancellation or terminal-state update 4. only then considering direct DB intervention If you want, I can next help you with one of these: - a read-only SQL checklist to identify orphaned runs - a Prefect API approach to update their state - a careful DB-edit approach after inspecting your schema first
k
@Marvin prefect api to update state
m
✅ thought for 132.6 seconds
Yes — you can update a stuck flow run state through the Prefect API. REST endpoint for flow runs
Copy code
text
POST /api/flow_runs/<FLOW_RUN_ID>/set_state
Recommended payload for a stuck/orphaned run Use 
force: true
so Prefect bypasses orchestration checks. Example marking it as `CANCELLED`:
Copy code
bash
curl -X POST "http://<your-prefect-url>/api/flow_runs/<FLOW_RUN_ID>/set_state" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {
      "type": "CANCELLED",
      "name": "Cancelled",
      "message": "Manually marked cancelled after infrastructure disappeared",
      "data": null,
      "state_details": {}
    },
    "force": true
  }'
You could also use terminal states like: - 
FAILED
- 
CRASHED
- 
COMPLETED
but for your case, 
CANCELLED
or 
CRASHED
is usually more appropriate. Example as `CRASHED`:
Copy code
bash
curl -X POST "http://<your-prefect-url>/api/flow_runs/<FLOW_RUN_ID>/set_state" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {
      "type": "CRASHED",
      "name": "Crashed",
      "message": "Infrastructure vanished before state update",
      "data": null,
      "state_details": {}
    },
    "force": true
  }'
Python client example This is often easier and safer:
Copy code
python
import asyncio
from prefect import get_client
from prefect.states import Cancelled

async def fix_run(flow_run_id: str):
    async with get_client() as client:
        result = await client.set_flow_run_state(
            flow_run_id=flow_run_id,
            state=Cancelled(message="Manually marked cancelled after pod deletion"),
            force=True,
        )
        print(result)

asyncio.run(fix_run("<FLOW_RUN_ID>"))
Important - Use 
force=True
for stuck runs - This updates the flow run state, not the underlying Kubernetes pod - If many runs are stuck, you may need to repeat for each affected run Task run endpoint If you also need it for task runs:
Copy code
text
POST /api/task_runs/<TASK_RUN_ID>/set_state
Docs - Prefect API client - Detect zombie flows If you want, I can help you with: 1. a script to find all stale 
RUNNING
/ 
CANCELLING
runs and mark them 
CRASHED
, or 2. the exact 
curl
command for your self-hosted URL format.
k
@Marvin Can i do it in the DB?
m
✅ thought for 8.7 seconds
You can, but I do not recommend it unless the API route is impossible. Direct DB edits in Prefect are risky because state is not just “one column update” in practice — there is state history and related orchestration behavior. If you write the wrong thing, you can leave the run in an inconsistent state. Best practice: use: - 
POST /api/flow_runs/<id>/set_state
- or the Python client with 
force=True
That said, if your system is too unhealthy for the API to work, then careful DB intervention can be a last resort. Important warning before any DB edit - take an RDS snapshot / DB backup first - test on one obviously orphaned run first - prefer updating only runs whose infrastructure is definitely gone - do not mass-update without confirming the schema in your DB Also, I want to be careful here: I should not guess the exact SQL mutation for your schema from memory. Prefect’s schema can vary by version, and I don’t want to give you unsafe SQL that corrupts state history. What I can safely recommend right now: 1. Inspect the schema first Run read-only queries like:
Copy code
sql
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'flow_run'
ORDER BY ordinal_position;
and:
Copy code
sql
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'flow_run_state'
ORDER BY ordinal_position;
Then also inspect one stuck run:
Copy code
sql
SELECT id, state_id, state_type, state_name, updated
FROM flow_run
WHERE id = '<FLOW_RUN_ID>';
and its state row:
Copy code
sql
SELECT *
FROM flow_run_state
WHERE id = (
  SELECT state_id
  FROM flow_run
  WHERE id = '<FLOW_RUN_ID>'
);
That will tell us exactly how your installation is storing the current state and state history. 2. Prefer inserting a new terminal state row over mutating old history In systems like Prefect, usually the safer pattern is: - insert a new state row - update 
flow_run.state_id
, 
flow_run.state_type
, 
flow_run.state_name
- possibly set 
end_time
/ 
updated
But again, I do not want to give you exact write SQL without verifying your schema. 3. If you want to proceed, I can help carefully If you paste the output of these read-only queries: - 
flow_run
columns - 
flow_run_state
columns - one example stuck run row - its current 
flow_run_state
row then I can help you craft the smallest possible, version-appropriate SQL update path. So short answer: Yes, but only as a last resort, and I recommend we inspect your schema first before writing any SQL.
k
@Marvin give me python sdk to mark them as canceled
m
✅ thought for 9.9 seconds
Here’s a simple Python script using the Prefect client to mark specific flow runs as 
Cancelled
with 
force=True
.
Copy code
python
import asyncio

from prefect import get_client
from prefect.states import Cancelled


FLOW_RUN_IDS = [
    "00000000-0000-0000-0000-000000000000",
    "11111111-1111-1111-1111-111111111111",
]


async def mark_runs_cancelled(flow_run_ids: list[str]) -> None:
    async with get_client() as client:
        for flow_run_id in flow_run_ids:
            result = await client.set_flow_run_state(
                flow_run_id=flow_run_id,
                state=Cancelled(
                    message="Manually marked cancelled after orphaned/stuck run"
                ),
                force=True,
            )
            print(f"{flow_run_id}: {result.status}")


if __name__ == "__main__":
    asyncio.run(mark_runs_cancelled(FLOW_RUN_IDS))
If you want to target all stale running/cancelling runs, here’s a version that first finds them, then marks them cancelled:
Copy code
python
import asyncio
from datetime import datetime, timedelta, timezone

from prefect import get_client
from prefect.client.schemas.filters import (
    FlowRunFilter,
    FlowRunFilterState,
    FlowRunFilterStateName,
)
from prefect.states import Cancelled


STALE_AFTER_HOURS = 1


async def mark_stale_runs_cancelled() -> None:
    cutoff = datetime.now(timezone.utc) - timedelta(hours=STALE_AFTER_HOURS)

    async with get_client() as client:
        flow_runs = await client.read_flow_runs(
            flow_run_filter=FlowRunFilter(
                state=FlowRunFilterState(
                    name=FlowRunFilterStateName(
                        any_=["Running", "Cancelling"]
                    )
                )
            )
        )

        for flow_run in flow_runs:
            if flow_run.updated and flow_run.updated < cutoff:
                result = await client.set_flow_run_state(
                    flow_run_id=flow_run.id,
                    state=Cancelled(
                        message="Manually marked cancelled because run was stale"
                    ),
                    force=True,
                )
                print(
                    f"{flow_run.id} ({flow_run.name}): {result.status}"
                )


if __name__ == "__main__":
    asyncio.run(mark_stale_runs_cancelled())
Before running, make sure the script can reach your Prefect API, for example with: - 
PREFECT_API_URL
- and any auth env vars if applicable Example:
Copy code
bash
export PREFECT_API_URL="http://<your-prefect-server>/api"
python mark_cancelled.py
Docs: - Prefect API client If you want, I can also give you: - a version that marks them 
Crashed
instead - a script that filters by deployment name - a script that does a dry run first
k
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://prefect-trn1.internal.usw1.fundamental-research.tech/api/flow_runs/e57e4d1d-70f9-4bf5-aba8-068a900a341a/set_state>' - Response: {'exception_message': 'Internal Server Error'} - For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500>
when using the api
@Marvin 
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://prefect-trn1.internal.usw1.fundamental-research.tech/api/flow_runs/e57e4d1d-70f9-4bf5-aba8-068a900a341a/set_state>' - Response: {'exception_message': 'Internal Server Error'} - For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500>
when using the api
m
✅ thought for 12.5 seconds
A 
500
on 
set_state
means the server itself failed while trying to process the state change. Given your earlier symptoms, the most likely reason is: the Prefect API is still hitting the same unhealthy Postgres/locking problem while processing the state update. So the API request is valid, but the server cannot complete it. What this means practically - your Python/client code is probably fine - the 
set_state
endpoint itself is failing inside Prefect - the real error will be in the Prefect server or background-services logs, not in the client traceback What to check next Look in the server API pod logs at the same timestamp as the 
500
. Search for: - 
set_state
- the specific flow run id - 
TimeoutError
- 
asyncpg
- 
sqlalchemy
- 
lock
- 
IntegrityError
Because Prefect usually logs the underlying stack trace there. Most likely causes 1. Postgres query timeout / lock wait during state transition 2. state transition trying to touch a stuck/orphaned run and blocking 3. broader DB saturation 4. transaction failure in the API service Since you already observed blocked queries, I would assume DB contention until proven otherwise. Recommendation If the API cannot mark the run cancelled because the database is too unhealthy, then you have two realistic options: Option 1 — stabilize DB first - reduce API replicas to 1 - reduce background service replicas to 1 - wait for DB pressure to drop - retry the API state update Option 2 — last resort DB cleanup If the API keeps returning 
500
, then API-based cleanup may not be possible until the DB is healthy again. At that point, careful DB intervention becomes the fallback. I know you asked earlier about DB updates — I avoided giving write SQL without schema verification. But now that the API is failing with 500, that fallback may be necessary if you need immediate cleanup. Safest next step before DB edits Please get: 1. the server log traceback for that exact 
500
2. the schema for 
flow_run
and 
flow_run_state
Read-only queries:
Copy code
sql
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'flow_run'
ORDER BY ordinal_position;
Copy code
sql
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'flow_run_state'
ORDER BY ordinal_position;
Copy code
sql
SELECT id, state_id, state_type, state_name, start_time, end_time, updated
FROM flow_run
WHERE id = 'e57e4d1d-70f9-4bf5-aba8-068a900a341a';
Copy code
sql
SELECT *
FROM flow_run_state
WHERE id = (
  SELECT state_id
  FROM flow_run
  WHERE id = 'e57e4d1d-70f9-4bf5-aba8-068a900a341a'
);
If you paste those results, I can help you determine the minimum-risk manual fix. Short answer The 
500
is probably not because you used the API wrong — it’s likely because the Prefect server cannot complete the state transition due to the same backend DB problem. If you want, send me the server-side traceback for that 
500
, and I can tell you whether it’s safe to keep trying the API or whether you should move to a DB-level workaround.
k
@Marvin
Copy code
prefect=> SELECT column_name, data_type
prefect-> FROM information_schema.columns
prefect-> WHERE table_name = 'flow_run'
prefect-> ORDER BY ordinal_position;
        column_name         |        data_type
----------------------------+--------------------------
 id                         | uuid
 created                    | timestamp with time zone
 updated                    | timestamp with time zone
 name                       | character varying
 state_type                 | USER-DEFINED
 run_count                  | integer
 expected_start_time        | timestamp with time zone
 next_scheduled_start_time  | timestamp with time zone
 start_time                 | timestamp with time zone
 end_time                   | timestamp with time zone
 total_run_time             | interval
 flow_version               | character varying
 parameters                 | jsonb
 idempotency_key            | character varying
 context                    | jsonb
 empirical_policy           | jsonb
 tags                       | jsonb
 auto_scheduled             | boolean
 flow_id                    | uuid
 deployment_id              | uuid
 parent_task_run_id         | uuid
 state_id                   | uuid
 state_name                 | character varying
 infrastructure_document_id | uuid
 work_queue_name            | character varying
 state_timestamp            | timestamp with time zone
 created_by                 | jsonb
 infrastructure_pid         | character varying
 work_queue_id              | uuid
 job_variables              | jsonb
 deployment_version         | character varying
 labels                     | jsonb
(32 rows)

prefect=> SELECT column_name, data_type
prefect-> FROM information_schema.columns
prefect-> WHERE table_name = 'flow_run_state'
prefect-> ORDER BY ordinal_position;
    column_name     |        data_type
--------------------+--------------------------
 id                 | uuid
 created            | timestamp with time zone
 updated            | timestamp with time zone
 type               | USER-DEFINED
 timestamp          | timestamp with time zone
 name               | character varying
 message            | character varying
 state_details      | jsonb
 data               | jsonb
 flow_run_id        | uuid
 result_artifact_id | uuid
(11 rows)

prefect=> SELECT id, state_id, state_type, state_name, start_time, end_time, updated
prefect-> FROM flow_run
prefect-> WHERE id = 'e57e4d1d-70f9-4bf5-aba8-068a900a341a';
                  id                  |               state_id               | state_type | state_name |          start_time           | end_time |            updated
--------------------------------------+--------------------------------------+------------+------------+-------------------------------+----------+-------------------------------
 e57e4d1d-70f9-4bf5-aba8-068a900a341a | 019edb50-9d3e-7a1a-bf70-162dc5f4160f | CANCELLING | Cancelling | 2026-06-18 14:39:23.499095+00 |          | 2026-06-18 15:19:13.988422+00
(1 row)

prefect=> SELECT *
prefect-> FROM flow_run_state
prefect-> WHERE id = (
prefect(>   SELECT state_id
prefect(>   FROM flow_run
prefect(>   WHERE id = 'e57e4d1d-70f9-4bf5-aba8-068a900a341a'
prefect(> );
                  id                  |            created            |            updated            |    type    |           timestamp           |    name    |              message               |                                                                                                                                                                                                                                                      state_details                                                                                                                                                                                                                                                       | data |             flow_run_id              | result_artifact_id
--------------------------------------+-------------------------------+-------------------------------+------------+-------------------------------+------------+------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+--------------------------------------+--------------------
 019edb50-9d3e-7a1a-bf70-162dc5f4160f | 2026-06-18 15:19:14.001011+00 | 2026-06-18 15:19:14.001018+00 | CANCELLING | 2026-06-18 15:19:13.982078+00 | Cancelling | The parent flow run was cancelled. | {"deferred": false, "cache_key": null, "pause_key": null, "retriable": null, "flow_run_id": "e57e4d1d-70f9-4bf5-aba8-068a900a341a", "task_run_id": "a0cfd915-b1f8-4921-a388-94fdccac4d7e", "traceparent": null, "pause_timeout": null, "refresh_cache": null, "transition_id": null, "scheduled_time": null, "cache_expiration": null, "pause_reschedule": false, "run_input_keyset": null, "child_flow_run_id": null, "task_parameters_id": null, "untrackable_result": false, "deployment_concurrency_lease_id": null} |      | e57e4d1d-70f9-4bf5-aba8-068a900a341a |
(1 row)
m
✅ thought for 11.5 seconds
Yes — with that schema, you can do a careful DB-side fix for this one stuck run. Since the run is clearly stuck in 
CANCELLING
, and the API is returning 
500
, the safest minimal intervention is: 1. insert a new terminal state into 
flow_run_state
2. point 
flow_run.state_id
to that new row 3. update 
flow_run.state_type
, 
state_name
, 
state_timestamp
, 
end_time
, 
updated
I would mark this one as CANCELLED. Important before you do this - take a DB snapshot / backup first - do this for one run only first - run inside a transaction Here is a careful SQL example for this exact flow run:
Copy code
sql
BEGIN;

WITH new_state AS (
  INSERT INTO flow_run_state (
    id,
    created,
    updated,
    type,
    timestamp,
    name,
    message,
    state_details,
    data,
    flow_run_id,
    result_artifact_id
  )
  VALUES (
    gen_random_uuid(),
    now(),
    now(),
    'CANCELLED',
    now(),
    'Cancelled',
    'Manually marked cancelled after orphaned/stuck run',
    jsonb_build_object(
      'deferred', false,
      'cache_key', null,
      'pause_key', null,
      'retriable', null,
      'flow_run_id', 'e57e4d1d-70f9-4bf5-aba8-068a900a341a',
      'task_run_id', null,
      'traceparent', null,
      'pause_timeout', null,
      'refresh_cache', null,
      'transition_id', null,
      'scheduled_time', null,
      'cache_expiration', null,
      'pause_reschedule', false,
      'run_input_keyset', null,
      'child_flow_run_id', null,
      'task_parameters_id', null,
      'untrackable_result', false,
      'deployment_concurrency_lease_id', null
    ),
    NULL,
    'e57e4d1d-70f9-4bf5-aba8-068a900a341a',
    NULL
  )
  RETURNING id, timestamp
)
UPDATE flow_run
SET
  state_id = new_state.id,
  state_type = 'CANCELLED',
  state_name = 'Cancelled',
  state_timestamp = new_state.timestamp,
  end_time = now(),
  updated = now()
FROM new_state
WHERE flow_run.id = 'e57e4d1d-70f9-4bf5-aba8-068a900a341a';

COMMIT;
Then verify:
Copy code
sql
SELECT id, state_id, state_type, state_name, start_time, end_time, updated
FROM flow_run
WHERE id = 'e57e4d1d-70f9-4bf5-aba8-068a900a341a';
and:
Copy code
sql
SELECT id, type, name, message, timestamp
FROM flow_run_state
WHERE flow_run_id = 'e57e4d1d-70f9-4bf5-aba8-068a900a341a'
ORDER BY timestamp DESC
LIMIT 5;
Notes - this preserves state history by adding a new state row instead of mutating the old one - it is safer than editing the existing 
CANCELLING
row in place - 
gen_random_uuid()
requires `pgcrypto`; if unavailable, use 
uuid_generate_v4()
if your DB has 
uuid-ossp
If you want to check first whether 
gen_random_uuid()
exists:
Copy code
sql
SELECT gen_random_uuid();
If that fails, try:
Copy code
sql
SELECT uuid_generate_v4();
If you want, next I can give you: 1. a version for multiple stuck flow runs, or 2. a read-only query to list all runs stuck in 
CANCELLING
for more than N minutes.
k
@Marvin i want to do this for all the pending cancelling workflows
m
✅ thought for 12.6 seconds
You can, but please be careful. Bulk DB updates are risky. Since your API is failing and you already have stuck 
CANCELLING
runs, the safest bulk approach is still: - insert a new 
CANCELLED
state row per stuck run - update each corresponding 
flow_run
to point to that new state Do not overwrite the old state rows in place. Before running this: Safety checklist - take an RDS snapshot first - test on 1–2 runs first - ideally pause/reduce Prefect activity while doing this - run it in a transaction - confirm 
gen_random_uuid()
exists Check UUID function:
Copy code
sql
SELECT gen_random_uuid();
If that fails, stop and tell me — I’ll adapt it for 
uuid_generate_v4()
. Preview which runs will be changed I recommend previewing first:
Copy code
sql
SELECT id, name, state_type, state_name, updated
FROM flow_run
WHERE state_type = 'CANCELLING'
ORDER BY updated ASC;
Bulk update all currently 
CANCELLING
runs to 
CANCELLED
 This version adds a fresh terminal state row for each run, then updates 
flow_run
.
Copy code
sql
BEGIN;

WITH target_runs AS (
  SELECT id
  FROM flow_run
  WHERE state_type = 'CANCELLING'
),
inserted_states AS (
  INSERT INTO flow_run_state (
    id,
    created,
    updated,
    type,
    timestamp,
    name,
    message,
    state_details,
    data,
    flow_run_id,
    result_artifact_id
  )
  SELECT
    gen_random_uuid(),
    now(),
    now(),
    'CANCELLED',
    now(),
    'Cancelled',
    'Manually marked cancelled after orphaned/stuck run',
    jsonb_build_object(
      'deferred', false,
      'cache_key', null,
      'pause_key', null,
      'retriable', null,
      'flow_run_id', tr.id,
      'task_run_id', null,
      'traceparent', null,
      'pause_timeout', null,
      'refresh_cache', null,
      'transition_id', null,
      'scheduled_time', null,
      'cache_expiration', null,
      'pause_reschedule', false,
      'run_input_keyset', null,
      'child_flow_run_id', null,
      'task_parameters_id', null,
      'untrackable_result', false,
      'deployment_concurrency_lease_id', null
    ),
    NULL,
    tr.id,
    NULL
  FROM target_runs tr
  RETURNING id, flow_run_id, timestamp
)
UPDATE flow_run fr
SET
  state_id = ist.id,
  state_type = 'CANCELLED',
  state_name = 'Cancelled',
  state_timestamp = ist.timestamp,
  end_time = COALESCE(fr.end_time, now()),
  updated = now()
FROM inserted_states ist
WHERE fr.id = ist.flow_run_id;

COMMIT;
Verify results Afterward:
Copy code
sql
SELECT state_type, state_name, count(*)
FROM flow_run
GROUP BY 1, 2
ORDER BY 3 DESC;
And specifically:
Copy code
sql
SELECT id, name, state_type, state_name, updated, end_time
FROM flow_run
WHERE state_type = 'CANCELLED'
ORDER BY updated DESC
LIMIT 50;
Safer variant: only stale cancelling runs If you want to avoid touching very recent cancellations still being processed, use only old ones, for example older than 30 minutes: ```sql BEGIN; WITH target_runs AS ( SELECT id FROM flow_run WHERE state_type = 'CANCELLING' AND updated < now() - interval '30 minutes' ), inserted_states AS ( INSERT INTO flow_run_state ( id, created, updated, type, timestamp, name, message, state_details, data, flow_run_id, result_artifact_id ) SELECT gen_random_uuid(), now(), now(), 'CANCELLED', now(), 'Cancelled', 'Manually marked cancelled after orphaned/stuck run', jsonb_build_object( 'deferred', false, 'cache_key', null, 'pause_key', null, 'retriable', null, 'flow_run_id', tr.id, 'task_run_id', null, 'traceparent', null, 'pause_timeout', null, 'refresh_cache', null, 'transition_id', null, 'scheduled_time', null, 'cache_expiration', null, 'pause_reschedule', false, 'run_input_keyset', null, 'child_flow_run_id', null, 'task_parameters_id', null, 'untrackable_result', false, 'deployment_concurrency_lease_id', null ),
NULL, tr.id, NULL FROM target_runs tr RETURNING id, flow_run_id, timestamp ) UPDATE flow_run fr SET state_id = ist.id, state_type = 'CANCELLED', state_name = 'Cancelled', state_timestamp = ist.timestamp, end_time = COALESCE(fr.end_time, now()), updated = now() FROM inserted_states ist WHERE fr.id = ist.flow_run_id; COMMIT; ``` My recommendation Use the stale-only version first, not “all cancelling”, unless you are sure the system is currently unhealthy enough that none of them will complete naturally. If you want, I can also give you: - a preview query showing exactly how many rows would be affected - a version for 
RUNNING
runs stuck longer than N hours - a rollback strategy if you need one
k
@Marvin after disabling sepearate background services deployment, it works fine. when am i supposed to enable it
m
✅ thought for 13.1 seconds
That’s a very useful result. It means your system is stable when Prefect runs in the simpler mode, and the problems start when you split out background services. So for now: you should leave separate background services disabled. You should enable a separate background-services deployment only when you actually need it and your infrastructure is ready for it. When to enable separate background services Typically when you want one or more of these: - scaling: isolate API traffic from background orchestration work - high availability: run background services independently of the API pod - operational isolation: tune/restart background services separately - larger self-hosted deployments: many deployments, many runs, lots of orchestration events When not to enable it yet Based on your experience, not yet if: - your Postgres is still sensitive to lock/read pressure - Redis/Docket setup is still being validated - you only have modest workload - one combined server process works fine In other words, separate background services is not something you must enable by default. It’s more of a scaling/architecture option. What your test tells us Since disabling it fixed the issue: - the extra concurrency/coordination introduced by the separate deployment was likely too much for your DB/setup - or the separate deployment added pressure through more background work than the DB could comfortably handle My recommendation Keep it disabled until all of these are true: 1. Stable Postgres - no recurring query blocking - no asyncpg timeouts - healthy CPU / memory / latency 2. Stable Redis config - external Redis/ElastiCache configured - cluster mode disabled - 
PREFECT_SERVER_DOCKET_URL
set correctly if/when needed 3. You actually need the separation For example: - you want to scale API separately from scheduling/cancellations/etc. - one combined deployment is becoming resource-constrained - you need more resilience around orchestration services 4. You re-enable gradually When you do re-enable it: - start with 1 background-services replica only - keep API replicas low at first - monitor Postgres closely - confirm no lock buildup Practical rule of thumb If your current setup works well with background services not separated, then that is the right setup for now. You are “supposed” to enable separate background services when: - the simple deployment becomes a bottleneck, or - you need operational separation for scale/HA —not just because the option exists. Docs: - Self-hosting Prefect If you want, I can help you with a decision checklist for: - combined server only vs - separate background services based on your run volume and deployment size.
k
@Marvin ok i will keep it disabled. what about redis then?
m
✅ thought for 7.5 seconds
If you’re not running background services as a separate deployment, then Redis becomes much less urgent. Short version: You may not need external Redis right now. Why: - the main reason Redis matters here is coordination for distributed background services - when you keep everything in the simpler combined setup, that coordination need is reduced - your earlier Redis 
ClusterCrossSlotError
came from the external cluster-mode Redis setup, which was adding trouble rather than helping What I recommend now If your current setup works with separate background services disabled: - keep the architecture simple - do not introduce cluster-mode ElastiCache - only add Redis if you have a specific feature/scale need that requires it If you do use Redis with Prefect self-hosted Use: - ElastiCache Redis/Valkey - cluster mode disabled - preferably a simple replication group Avoid: - cluster mode enabled Because that is what triggered the cross-slot problem. So should you remove Redis entirely? That depends on how your self-hosted configuration is currently wired. If your deployment is functioning fine without the separate background-services architecture, then the best approach is usually: - keep Redis configuration minimal - or remove external Redis dependency if it is not required by the features you are using But if you are using Redis-backed messaging/events features in your setup, then keep Redis — just use the right topology. Safe recommendation for you now - Do not use ElastiCache cluster mode enabled - if Redis is needed, use cluster mode disabled - if your deployment works without external Redis complexity, keep it simple for now So the practical answer is: For your current stable setup, Redis is optional unless you specifically need the Redis-backed services/features. If you do use it, use non-cluster ElastiCache only. If you want, I can help you determine whether your current Prefect config actually requires Redis at all based on your env vars / Helm values.
k
@Marvin i have hpa enabled for the server
m
✅ thought for 7.1 seconds
With HPA enabled on the server, Redis becomes more important again. Why: - if the 
server
pods scale horizontally, you now have multiple Prefect server instances - multiple instances can run/coordinate background work - that’s exactly where distributed coordination matters - without proper shared coordination, you risk duplicate or conflicting background activity So if you keep HPA on the server, I would not rely on an in-memory/no-coordination approach. Recommendation for your setup Since: - separate background services disabled worked better - but server HPA is enabled then the best fit is usually: - keep background services not separate - use external Redis - set 
PREFECT_SERVER_DOCKET_URL
- use ElastiCache cluster mode disabled That gives multiple server pods a shared coordination backend without introducing the Redis Cluster cross-slot problem. What Redis should be Use: - ElastiCache Redis/Valkey - cluster mode disabled - TLS if available - one primary endpoint Example:
Copy code
text
PREFECT_SERVER_DOCKET_URL=<rediss://default:<AUTH_TOKEN>@<redis-primary-endpoint>:6379/0>
If you are also using Redis-backed messaging/events, you may also configure:
Copy code
text
PREFECT_REDIS_MESSAGING_URL=<rediss://default:<AUTH_TOKEN>@<redis-primary-endpoint>:6379/1>
But the key one for multi-server coordination is:
Copy code
text
PREFECT_SERVER_DOCKET_URL
Important Do not use: - ElastiCache cluster mode enabled because that caused your earlier:
Copy code
text
ClusterCrossSlotError
Practical guidance If HPA can scale the server above 
1
replica, then yes, I would use external non-cluster Redis. If you want to avoid Redis entirely, then the safer path is: - disable HPA - keep a single server replica So the tradeoff is: - single server replica: simpler, may avoid Redis - multiple server replicas via HPA: use external non-cluster Redis If you want, I can help you produce the exact env vars / Helm values for: 1. HPA + combined server + external ElastiCache non-cluster, or 2. single replica without HPA.
k
@Marvin the flow appears to bein the runnign state when OOM? how to mark it as crashed if OOM?
m
✅ thought for 139.8 seconds
Yes — this is a common pattern when the infrastructure gets OOM-killed but Prefect never receives a clean terminal update. For Prefect 3.x, the recommended way to mark OOM-killed runs as 
Crashed
automatically is: 1. Use flow heartbeats Prefect 3.x supports flow-run heartbeats. If a run stops heartbeating because the pod was OOM-killed, you can detect that. Important setting:
Copy code
text
PREFECT_FLOWS_HEARTBEAT_FREQUENCY
Default in newer Prefect 3.x is typically 
180
seconds. Example:
Copy code
bash
PREFECT_FLOWS_HEARTBEAT_FREQUENCY=180
2. Create an automation that marks missing-heartbeat runs as 
Crashed
 For self-hosted Prefect, this is the main pattern for “zombie” flows. Docs: - Detect zombie flows A typical approach is: - watch for 
prefect.flow-run.heartbeat
- if no follow-up activity appears within a time window - change the flow run state to 
CRASHED
Example Python automation:
Copy code
python
from datetime import timedelta

from prefect.automations import Automation
from prefect.client.schemas.objects import StateType
from prefect.events.actions import ChangeFlowRunState
from prefect.events.schemas.automations import EventTrigger, Posture
from prefect.events.schemas.events import ResourceSpecification

automation = Automation(
    name="Crash zombie flows",
    trigger=EventTrigger(
        after={"prefect.flow-run.heartbeat"},
        expect={"prefect.flow-run.*"},
        match=ResourceSpecification(
            {"prefect.resource.id": ["prefect.flow-run.*"]}
        ),
        for_each={"prefect.resource.id"},
        posture=Posture.Proactive,
        threshold=1,
        within=timedelta(seconds=540),
    ),
    actions=[
        ChangeFlowRunState(
            state=StateType.CRASHED,
            message="Flow run marked as crashed due to missing heartbeats.",
        )
    ],
)

automation.create()
That 
540
seconds is based on 
3 x 180s
heartbeats. 3. For Kubernetes OOM specifically OOM usually shows up as process exit code 
137
/ 
SIGKILL
. If the worker fully observes the infrastructure exit, it may already mark the run as crashed. But if the pod disappears or the worker misses it, the heartbeat automation is the more reliable fallback. 4. Also fix the root cause If runs are OOMing, increase memory in your work pool / Kubernetes job spec. Practical recommendation for you Since you are self-hosted and already saw stuck 
Running
runs after pod loss, I would do both: - set 
PREFECT_FLOWS_HEARTBEAT_FREQUENCY=180
- create the zombie-flow automation above That way OOM-killed pods that never report back will get marked 
Crashed
automatically. Docs - Detect zombie flows - Worker healthchecks - Infrastructure debugging If you want, I can also give you: 1. a version of that automation for your exact self-hosted API URL/setup, or 2. a one-off script to mark currently stuck 
Running
runs as 
Crashed
.
2 Views
Open in Slack
PreviousNext
Powered by