https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> My deployment has status "Not ready"....
# ask-marvin
r
@Marvin My deployment has status "Not ready". Why ?
m
✅ thought for 100.7 seconds
In Prefect 3.x, 
Not ready
usually means no worker is actively polling the deployment's work pool / work queue. A few common reasons: - The deployment was just created New deployments default to 
Not ready
until something starts polling them. - No worker is running for that work pool If nothing is listening for scheduled work, the deployment stays 
Not ready
. - The worker stopped, crashed, or lost connectivity Prefect marks deployments 
Not ready
if they haven't been polled recently. - The associated work queue / pool is unhealthy or idle Deployment readiness depends on the queue/pool being actively polled too. What to check 1. Confirm the deployment's work pool. 2. Make sure a worker is running for that pool. 3. Check whether the worker is continuously connected and polling. 4. Verify the pool/queue isn't paused or misconfigured. If you're using the CLI, I can help you check those next, but the main command to run on your infra is:
Copy code
bash
prefect worker start --pool <your-pool-name>
If that worker is already running, then the next thing to inspect is whether it's polling successfully or erroring. Useful docs: - Workers and work pools - Deployments overview If you want, send me: - your deployment's work pool name - whether you're using schedules - whether a worker is currently running and I can help narrow down the exact cause.
r
@Marvin I am using process worker. Worker is running, I see status Ready is Work Pools/default_pool page. Name of my pool is default_pool. Also I am able to run flow from UI. My deployment looks like :
Copy code
RunnerDeployment(
        name="scanner1",
        flow_name="simple_scanner",
        entrypoint="scanner.main:simple_scanner",
        work_pool_name="default_pool",
        storage=LocalStorage(path=env_config.working_directory),
        tags=["scanner", "orchestrator"],
        description=description,
        version=get_version("scanner"),
        schedules=[
            DeploymentScheduleCreate(
                schedule=IntervalSchedule(interval=timedelta(minutes=5)),
                active=True,
                max_scheduled_runs=1, 
            )
        ],
        concurrency_limit=1, 
        concurrency_options=ConcurrencyOptions(
            collision_strategy=ConcurrencyLimitStrategy.CANCEL_NEW, 
        ),
m
✅ thought for 172.6 seconds
Thanks — that extra detail helps. Given what you described, this is probably not a real execution problem. It’s most likely one of these two things: 1. The deployment status is just stale / misleading There’s an existing issue about deployments showing 
Not Ready
 even when workers are polling and runs succeed: - Confusing deployment status in the UI 2. Your deployment is associated with a queue/polling pattern that is not updating readiness the way the UI expects This can happen even while: - the work pool shows 
Ready
- the worker is online - manual runs from the UI succeed That’s because deployment readiness is not the same thing as “can this run execute?”. In practice: - manual runs succeeding means your worker and infrastructure are basically fine - pool = Ready means at least one worker is online for that pool - deployment = Not ready means Prefect does not think that deployment has been polled recently enough, or the readiness indicator wasn't updated correctly Your config looks normal at a glance: - 
work_pool_name="default_pool"
- interval schedule every 5 min - process worker - local storage So the next question is: Are scheduled runs actually being created and picked up every 5 minutes? If yes, then this is almost certainly just a readiness-status/UI issue, not a deployment problem. What I’d check next: 1. Does the Runs tab show new scheduled runs every 5 minutes? - If yes, scheduler is working. 2. Are those scheduled runs executed by the worker? - If yes, everything important is working. 3. Check whether the deployment has a specific work queue assigned - If the worker polls one queue and the deployment targets another, readiness can look wrong. If you want to inspect that from Python / CLI, I can help with exact commands. A useful inspect is:
Copy code
bash
prefect deployment inspect "simple_scanner/scanner1"
That will let us confirm: - work pool - work queue - status - schedule config A subtle thing I also want to call out: you’re showing a 
RunnerDeployment(...)
object. In Prefect, 
RunnerDeployment
is also used in runner/serve-related code paths, and those can behave a bit differently from normal worker-polled deployments. If you share how you create/apply this deployment — for example whether you are using: - 
flow.deploy(...)
- 
flow.serve(...)
- 
serve(...)
- direct client calls - 
prefect deploy
—I can tell you whether that object type is the reason for the status mismatch. So my current best read is: If scheduled runs are being created and executed, you can safely treat 
Not ready
as cosmetic here. If you want, send me: - the output of 
prefect deployment inspect "simple_scanner/scanner1"
- how you register/deploy this 
RunnerDeployment
and I’ll pinpoint the exact reason.
r
@Marvin Scheduled runs are not created - this is my main issue. prefect deployment inspect "simple_scanner/scanner1" :
Copy code
{
    'id': 'df112cb2-faa7-4acb-a719-0e38bc9ddf11',
    'created': '2026-06-17T09:18:33.984480Z',
    'updated': '2026-06-17T09:18:33.964484Z',
    'name': 'scanner1',
    'version': '0.0.0',
    'version_id': None,
    'version_info': None,
    'branch': None,
    'base': None,
    'root': None,
    'description': '\n# Popis\nPackage `scanner` obsahuje orchestration flow-y pre spustanie scannerov.\nNejde o
klasicku implementaciu jedneho LIFu, ale o spolocny entrypoint\npre scanner workload-y, ktore nasledne triggeruju
cielove LIF deploymenty.\n\n# Vstupne parametre\nHlavny flow prijima parameter `dry_run: bool`, ktory pri hodnote
`True`\nnevykona realne triggerovanie deploymentov a vrati iba scan vysledok.\n\n# Konfiguracia\nKonfiguracia scannera
je v `prefect_lifs.scanner.simple_scan_config`.\n\n# Zotavovanie\nAktualna implementacia podporuje zotavenie suborov,
ktore boli premenenovane, ale zlyhalo zalozenie flow.\n',
    'flow_id': '35b5b076-ab9a-47e9-a65e-135c424b48d1',
    'concurrency_limit': None,
    'global_concurrency_limit': {
        'id': '5a062b02-f824-49d4-85d7-514073c1ffea',
        'created': '2026-06-17T09:18:34.039602Z',
        'updated': '2026-06-17T09:39:53.481677Z',
        'active': True,
        'name': 'deployment:df112cb2-faa7-4acb-a719-0e38bc9ddf11',
        'limit': 1,
        'active_slots': 0,
        'slot_decay_per_second': 0.0
    },
    'concurrency_options': {'collision_strategy': 'CANCEL_NEW', 'grace_period_seconds': None},
    'paused': False,
    'schedules': [
        {
            'id': 'bcc64345-cab3-42c4-aa1c-57b8a68307c7',
            'created': '2026-06-17T09:18:34.015617Z',
            'updated': '2026-06-17T09:18:34.015630Z',
            'deployment_id': 'df112cb2-faa7-4acb-a719-0e38bc9ddf11',
            'schedule': {'interval': 300.0, 'anchor_date': '2026-06-17T09:18:33.743078Z', 'timezone': 'UTC'},
            'active': True,
            'max_scheduled_runs': None,
            'parameters': {},
            'slug': None
        }
    ],
    'job_variables': {},
    'parameters': {},
    'pull_steps': [{'prefect.deployments.steps.set_working_directory': {'directory': '/apphome/clag'}}],
    'tags': ['scanner', 'orchestrator'],
    'labels': {'prefect.flow.id': '35b5b076-ab9a-47e9-a65e-135c424b48d1'},
    'work_queue_name': 'default',
    'last_polled': None,
    'parameter_openapi_schema': {
        'type': 'object',
        'title': 'Parameters',
        'properties': {
            'history_days': {'type': 'integer', 'title': 'history_days', 'default': 14, 'position': 0},
            'artifact_key_prefix': {
                'type': 'string',
                'title': 'artifact_key_prefix',
                'default': 'simple-scanner',
                'position': 1
            },
            'artifact_key_interval': {
                'type': 'integer',
                'title': 'artifact_key_interval',
                'default': 24,
                'position': 2
            }
        }
    },
    'path': None,
    'entrypoint': 'scanner.main:simple_scanner',
    'storage_document_id': None,
    'infrastructure_document_id': None,
    'created_by': None,
    'updated_by': None,
    'work_queue_id': 'bb3b38f1-a1c5-49c2-9ab5-d1d9deeaeda7',
    'enforce_parameter_schema': True,
    'work_pool_name': 'default_pool',
    'status': 'NOT_READY',
    'automations': []
}
m
✅ thought for 190.7 seconds
Thanks — this narrows it down a lot. The key detail is this:
Copy code
'last_polled': None
and
Copy code
'status': 'NOT_READY'
That explains why the worker has never polled this deployment's queue, but it does not explain why scheduled runs are not being created. In Prefect 3.x, scheduled run creation is handled by the scheduler service, not by the worker. So your issue is most likely: the scheduler service is not running / not enabled because your deployment itself looks schedulable: - 
paused: False
- schedule exists - schedule is 
active: True
- 
work_queue_id
is set - 
work_pool_name
is set So the deployment config looks okay. Important distinction - Worker polls and executes runs - Scheduler service creates future scheduled flow runs - Foreman updates 
READY
/ 
NOT_READY
So right now you may actually have two separate issues: 1. Worker polling/readiness issue since 
last_polled
is 
None
2. Scheduler service issue since no scheduled runs are being created But the lack of scheduled runs points more strongly to the scheduler service than to the worker. What to check next If you're running Prefect server / self-hosted, check whether scheduler services are enabled. Look for these settings / env vars:
Copy code
bash
PREFECT_API_SERVICES_SCHEDULER_ENABLED
It should effectively be 
true
. Also check the scheduler logs from your API/server process. You should see periodic scheduler activity. If you are in a containerized/self-hosted setup, confirm the API process is actually running background services. A useful thing to verify is whether any scheduled deployments in your environment are generating scheduled runs, or just this one. Why I think this is the scheduler service Your inspect output shows: - deployment is not paused - schedule is active - interval schedule is valid - no obvious bad config on the deployment So if zero scheduled runs are appearing, the scheduler likely isn't creating them. One more thing about your deployment creation path You are constructing a 
RunnerDeployment(...)
directly. That is a bit unusual unless you are intentionally using runner APIs. If you're mixing runner-style deployment objects with worker-based execution, that can create confusing behavior. So I want to confirm: how are you registering this deployment? For example, are you doing something like: - 
client.create_deployment(...)
- 
RunnerDeployment(...)
+ some apply call - 
flow.deploy(...)
- 
flow.serve(...)
That matters a lot here. Most likely diagnosis right now - No scheduled runs created → scheduler service problem - 
last_polled: None
 → worker has not actually polled the queue this deployment is attached to, despite the pool showing ready Those are related symptoms, but the first one is the blocker. Recommended next step Please send: 1. How you start Prefect server/API 2. How you start the process worker 3. The code/path you use to register this 
RunnerDeployment
If you want, I can also help you run a very short checklist to confirm the scheduler service is enabled in your environment.
r
@Marvin Prefect API server and Prefect background services are deployed in OCP cluster. I have restarted Background services and now all deployments have status "Not Ready".
m
✅ thought for 12.3 seconds
That strongly suggests the root problem is in the background services deployment, not in your deployment definition. If after restarting background services all deployments became 
Not Ready
, that usually means: - workers are not successfully polling anymore, and/or - scheduler/foreman services in the background-services pod are unhealthy or misconfigured Since earlier you also had no scheduled runs being created, the most likely explanation is: your background services are not actually running correctly after restart In self-hosted Prefect, those services are what handle things like: - scheduling future runs - readiness transitions - other server-side maintenance loops And 
Not Ready
for all deployments after the restart is a big signal that this is systemic, not deployment-specific. What to check immediately 1. Background services pod logs Look for errors from: - scheduler - foreman - database connectivity - leadership / startup issues - crashes / restart loops 2. API server logs Check whether API can talk to the database and whether background tasks are being enqueued. 3. Worker logs Even if the pool UI says Ready, confirm the worker is actually polling and not failing on each poll. 4. Database connectivity from background services If background services can’t reach the DB, they may start but do nothing useful. Why all deployments became 
Not Ready
 Prefect marks deployments 
Not Ready
when they have not been polled recently enough. If something in the system that updates readiness or supports polling is broken after restart, they can all drift into 
Not Ready
. Also, if scheduled runs are no longer being created, that reinforces that the services pod is unhealthy. Most likely failure modes in OCP/self-hosted - background-services pod started with wrong env/config after restart - DB connection issue - API URL / service URL mismatch - service account / network policy issue - multiple replicas / task queue issue - container is up but scheduler loops are failing internally What I’d inspect in logs Search for terms like:
Copy code
text
scheduler
foreman
database
sqlalchemy
connection refused
migration
leader
task queue
exception
traceback
Very important note If you're running multiple API/background replicas in self-hosted Prefect, there are known areas around background task queue behavior and readiness updates that can be tricky: - Redis-backed task queue work for multi-replica server deployments - Deployment readiness deadlock issue Those may or may not apply to you, but if this is HA/multi-replica OCP, it is relevant. My current best diagnosis Because: - scheduled runs were not being created - now all deployments are 
Not Ready
after background-services restart the issue is almost certainly platform-side, specifically your self-hosted Prefect server/background-services health, not the 
RunnerDeployment
config. If you paste: - background services logs after restart - number of API replicas / background-service replicas - your Prefect version I can help pinpoint whether this is: - scheduler not running - foreman failing - DB/task queue issue - HA/multi-replica problem.
r
@Marvin I have lot of deadlocks in bacground services log :
Copy code
sqlalchemy.exc.DBAPIError: (sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.DeadlockDetectedError'>: deadlock detected
DETAIL:  Process 116791 waits for ShareLock on transaction 11489556; blocked by process 116798.
Process 116798 waits for ShareLock on transaction 11489557; blocked by process 116791.
HINT:  See server log for query details.
[SQL: DELETE FROM event_resources WHERE event_resources.id IN (SELECT event_resources.id 
FROM event_resources 
WHERE event_resources.occurred < $1::TIMESTAMP WITH TIME ZONE 
 LIMIT $2::INTEGER) RETURNING event_resources.id]
[parameters: (DateTime(2026, 6, 15, 10, 29, 4, 756911, tzinfo=Timezone('UTC')), 200)]
(Background on this error at: <https://sqlalche.me/e/20/dbapi>)
m
✅ thought for 226.8 seconds
Yes — this is very likely the reason. That deadlock is in background event cleanup:
Copy code
sql
DELETE FROM event_resources
...
WHERE event_resources.occurred < ...
LIMIT 200
So your background services are getting stuck/failing while doing DB maintenance work against PostgreSQL. Why this matters to your symptoms If background services are repeatedly hitting DB deadlocks, they can fail to keep up with core server-side loops, including things that affect: - scheduled run creation - readiness updates - general background maintenance That matches what you observed: - scheduled runs not being created - after restart, deployments drift to 
Not Ready
- lots of deadlocks in background-services logs So this is not a problem with your deployment definition anymore — it’s a self-hosted Prefect/Postgres background services issue. What the deadlock is telling us The failing query is part of event cleanup / vacuum-style retention work. In a multi-replica or highly concurrent environment, multiple services can try to delete overlapping rows and PostgreSQL deadlocks them. There is also a related Prefect issue around readiness-update deadlocks: - Deployment readiness deadlock issue Your exact deadlock is on 
event_resources
, which suggests event cleanup contention, not just deployment readiness. Most likely environment factor If you have multiple background-service replicas, that makes this much more likely. Can you confirm: - how many background-services replicas you run? - how many API replicas you run? Because if more than one background-services pod is doing the same cleanup loop, that is a prime suspect. Immediate mitigations to try 1. Run only one background-services replica If you currently have multiple replicas, scale background services down to 
1
. 2. Reduce DB vacuum cleanup aggressiveness If you can set env vars for the background services pod, try making cleanup less aggressive:
Copy code
bash
PREFECT_SERVER_SERVICES_DB_VACUUM_BATCH_SIZE=50
PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS=7200
3. Check whether your Prefect version includes recent fixes There appears to be recent work around deadlocks and maintenance behavior, so version matters a lot here. 4. Inspect the size of the events tables If 
events
/ 
event_resources
have grown large, cleanup gets more expensive and deadlocks/timeouts get more likely. 5. Check for missing index on 
event_resources.occurred
 There has been recent work related to this area. A missing index would make cleanup much worse. My strongest recommendation right now If this is OCP and HA-style deployment: - temporarily scale background services to 1 replica - restart that single pod - see whether: - scheduled runs start being created again - deployments recover from 
Not Ready
- deadlocks stop That is the fastest way to validate whether replica contention is the main issue. Why I think this is the right next step Your deadlock is not from user flow execution; it’s from server maintenance. And your symptoms are system-wide. That combination usually points to background services concurrency / DB maintenance contention. Relevant links - Deployment readiness deadlock issue - SQLAlchemy DBAPI deadlock background info If you want, send me: - your Prefect version - number of API replicas - number of background-services replicas and I can tell you whether your topology is likely causing this and what the safest configuration is.
r
@Marvin I have 1 pod for background and 3 pods for API. I have scaled down everything to 0 waited 5 minutes and then started services (1 pod) and API (1 pod). Now all deplyments are in ready state. But what was the root cause ?
m
✅ thought for 15.5 seconds
Most likely root cause: a PostgreSQL deadlock storm in Prefect background services / API background tasks, which left server-side maintenance unhealthy enough that: - scheduled runs were not being created - deployments drifted to 
Not Ready
 - restarting only background services was not enough because stale concurrent work / locks / backlog likely remained - after fully scaling everything to 0 and bringing it back with a single API + single background pod, the contention cleared Given your topology, I’d separate this into direct cause and underlying cause. Direct cause Your Prefect control plane got into a bad state due to repeated DB deadlocks, especially around event cleanup like:
Copy code
sql
DELETE FROM event_resources
WHERE event_resources.occurred < ...
LIMIT 200
That means the server’s background maintenance work was contending in PostgreSQL and failing. Once that happens repeatedly, core services can fall behind. Why that affected deployments and schedules In Prefect 3.x: - scheduler creates scheduled runs - foreman/readiness logic updates deployment readiness - API/background task paths also do DB work related to events and internal bookkeeping If those services are repeatedly failing, blocked, or backlogged because of deadlocks, you can see exactly your symptoms: - no new scheduled runs - deployments showing 
Not Ready
- control plane looks partially alive, but orchestration is degraded Why scaling API from 3 to 1 helped Even with only 1 background-services pod, you had 3 API pods. Prefect server work is not isolated only to the background-services pod; some background-task behavior and readiness/event-related writes can still be triggered from API-side request handling. There is active evidence in Prefect issues/research that concurrent readiness/background updates across multiple server replicas can deadlock: - Deployment readiness deadlock issue So the likely sequence is: 1. multiple API replicas and/or background services were issuing overlapping DB work 2. PostgreSQL deadlocks started occurring 3. maintenance loops and orchestration tasks fell behind 4. schedules stopped being materialized 5. deployments became 
Not Ready
6. full scale-down cleared in-flight contention/backlog 7. single-replica restart removed the concurrency pressure, so system recovered So what was the real root cause? My best concise answer is: Concurrent server-side database operations in your self-hosted Prefect deployment caused PostgreSQL deadlocks, which degraded orchestration services. More specifically, the deadlocks you saw in 
event_resources
cleanup are a strong indicator that the control plane was unhealthy due to DB contention, not due to your deployment config. Why not your deployment config? Because once you restarted cleanly with 1 API + 1 background pod: - deployments became 
Ready
- that means worker/pool/deployment wiring was basically okay - the failure was systemic, not per-deployment Most likely contributing factors - multiple API replicas - Postgres contention on event cleanup / maintenance tables - possibly a backlog in events tables - possibly a Prefect version affected by known concurrency issues What I would do next to prevent recurrence 1. Check your Prefect version Some concurrency/deadlock fixes are recent. 2. Be cautious with multi-replica API on self-hosted Prefect Especially if you don’t have supporting architecture/settings for shared background task behavior. 3. Monitor Postgres for deadlocks and long-running deletes Especially on: - 
event_resources
- 
events
- deployment readiness-related updates 4. Inspect event table growth If those tables are large, cleanup gets more expensive and contention worsens. 5. If you scale API back to 3, watch carefully If deadlocks return, that’s strong confirmation replica concurrency is the trigger. Bottom line
The root cause was database deadlock contention in the Prefect control plane, not the deployment itself. Your full shutdown cleared the bad state, and running with 1 API + 1 background pod removed the concurrency pattern that was likely triggering it. If you send me your Prefect version, I can tell you whether it lines up with known deadlock issues and whether upgrading is likely the right fix.
r
@Marvin I am on 3.7.1
m
✅ thought for 8.0 seconds
That makes the likely answer stronger: you’re probably hitting a Prefect 3.7.1 self-hosted concurrency/deadlock issue, not a deployment misconfiguration. A few points: - You saw Postgres deadlocks in control-plane maintenance queries - scheduled runs stopped being created - all deployments became 
Not Ready
 - full reset + running 1 API / 1 background pod fixed it That pattern is very consistent with server-side contention between replicas / background tasks. There is a relevant Prefect issue on readiness deadlocks: - Deployment readiness deadlock issue And your deadlock in 
event_resources
suggests a related class of DB-concurrency problems in background maintenance/event cleanup. My best root-cause statement for 3.7.1 In Prefect 3.7.1, your self-hosted HA-ish control plane likely experienced database deadlocks between concurrent server processes (API replicas and/or background maintenance tasks). Those deadlocks prevented orchestration services from keeping up, which caused: - missed scheduled run creation - deployments remaining or becoming 
Not Ready
Why it recovered when you reduced replicas Because reducing to: - 
1
API pod - 
1
background-services pod removes a lot of concurrent DB activity and lock contention. So if you scale back up and the problem returns, that would strongly confirm it. What I recommend now 1. Stay on 1 API + 1 background pod for stability until you test further. 2. Watch logs for any new deadlocks before scaling up again. 3. Consider upgrading if a newer 3.x release is available with fixes in this area. I can also check the latest release notes for you to see whether there are likely fixes after 3.7.1.
r
@Marvin I have new details. Deadlock happens always during events vacuuming :
Copy code
11:29:05.911 | ERROR   | docket.worker - ↩ [  1538ms] vacuum_events_with_retention_overrides(){db-vacuum:retention-overrides}
Traceback (most recent call last):
  File "/home/prefect/venv/lib64/python3.12/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 550, in _prepare_and_execute
    self._rows = deque(await prepared_stmt.fetch(*parameters))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prefect/venv/lib64/python3.12/site-packages/asyncpg/prepared_stmt.py", line 177, in fetch
    data = await self.__bind_execute(args, 0, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prefect/venv/lib64/python3.12/site-packages/asyncpg/prepared_stmt.py", line 268, in __bind_execute
    data, status, _ = await self.__do_execute(
                      ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prefect/venv/lib64/python3.12/site-packages/asyncpg/prepared_stmt.py", line 257, in __do_execute
    return await executor(protocol)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "asyncpg/protocol/protocol.pyx", line 205, in bind_execute
asyncpg.exceptions.DeadlockDetectedError: deadlock detected
DETAIL:  Process 120238 waits for ShareLock on transaction 11490106; blocked by process 120236.
Process 120236 waits for ShareLock on transaction 11490105; blocked by process 120238.
HINT:  See server log for query details.
After this error deployment scheduler stop working and all deployments are switched to "Not Ready" state. And it never recoveres. Why ?
m
✅ thought for 180.9 seconds
Yes — this behavior makes sense given how Prefect’s background services work. Short version: The deadlock during 
vacuum_events_with_retention_overrides
likely exhausts or wedges the shared DB work used by background services, so: - the scheduler stops creating runs - deployments eventually become 
Not Ready
 - the system does not self-recover because the stuck background task never cleanly finishes and the process keeps a bad state until restart Why this happens That vacuum task is part of Prefect’s perpetual background services. When it deadlocks in PostgreSQL, the task can get stuck long enough to break the service loop’s ability to keep up with other work. In your case the sequence is likely: 1. 
vacuum_events_with_retention_overrides
starts 2. PostgreSQL deadlock occurs on event cleanup 3. background worker/task execution gets stuck or repeatedly fails in that area 4. scheduler stops successfully doing its loop 5. no scheduled runs are created 6. workers stop seeing fresh orchestrated work / readiness bookkeeping stops updating 7. deployments age into 
Not Ready
8. it never recovers because the unhealthy background task loop / DB state persists until process restart Why “never recovers”? Because this is not just a one-off failed scheduled run creation. It’s a control-plane service failure. In Prefect 3.x, background services are long-running “perpetual” loops. If one of the critical loops gets into a bad DB state, there isn’t always a clean circuit-breaker that fully resets it. So the process can stay alive but orchestration is effectively degraded. That matches your observation exactly: - one vacuum deadlock - then scheduler stops - then all deployments go 
Not Ready
- no automatic recovery Why vacuuming events can affect scheduling at all Because these services share the same Prefect server/database machinery. If event vacuuming is repeatedly deadlocking, it can block or starve other background DB operations. So while the failing query is “just” event cleanup, the effect is systemic. What this means for root cause Your real root cause is: In Prefect 3.7.1, a PostgreSQL deadlock during event vacuuming can destabilize self-hosted background services badly enough that scheduler/readiness processing stops and does not recover automatically. That is much more precise than “deployment Not Ready”. What I recommend 1. Upgrade beyond 3.7.1 if possible There are active fixes in this area and 3.7.1 is likely too old for some of them. 2. Keep API/background replicas minimal while testing Since you recovered with 1 API + 1 background pod, keep that topology until stable. 3. Treat event vacuum deadlocks as critical They are not harmless cleanup errors in your environment; they are causing orchestration failure. 4. Open/track a GitHub issue if you need a durable fix path There’s a related readiness deadlock issue: - Deployment readiness deadlock issue 5. Monitor background logs specifically for 
vacuum_events_with_retention_overrides
 That appears to be your earliest indicator before the scheduler dies. Best practical conclusion The reason it “never recovers” is that the deadlocked vacuum task leaves the background orchestration subsystem in a persistently unhealthy state, so scheduler and readiness loops stop functioning until the server processes are restarted. If you want, I can help you turn this into: - a concise GitHub bug report - or a short mitigation checklist for your OCP deployment.
r
@Marvin This delete :
Copy code
DELETE FROM event_resources WHERE event_resources.id IN (SELECT event_resources.id
FROM event_resources
WHERE event_resources.event_id IN (SELECT events.id
FROM events
WHERE events.event = $1::VARCHAR AND events.occurred < $2::TIMESTAMP WITH TIME ZONE)
is started from background services and can be started within 2 session in parallel ? Because error message suggests, that there are 2 dabase sessions (120236 and 120238):
Copy code
Process 120238 waits for ShareLock on transaction 11490106; blocked by process 120236
Or deadlock can be caused by delete and som another process ?
m
✅ thought for 13.8 seconds
Yes — the deadlock does not require two copies of the same vacuum task, and in your setup with 1 background pod + 3 API pods, it is very plausible that: - one session is the background vacuum delete, and - the other session is some other concurrent event-related DB operation from API/background processing So the answer is: It can be caused by two vacuum sessions in parallel, but it can also absolutely be caused by vacuum versus another process. In your case, given you had only 1 background-services pod, I would not assume “two background vacuum tasks” first. I would assume: vacuum delete collided with another transaction touching 
events
/ 
event_resources
. Why the SQL suggests that Your delete is:
Copy code
sql
DELETE FROM event_resources
WHERE event_resources.id IN (
  SELECT event_resources.id
  FROM event_resources
  WHERE event_resources.event_id IN (
    SELECT events.id
    FROM events
    WHERE events.event = $1
      AND events.occurred < $2
  )
)
That query: - reads from 
events
- reads from 
event_resources
- deletes from 
event_resources
So PostgreSQL may need locks involving both tables and the transactions referenced by those rows. If another transaction is, for example: - inserting into 
events
- inserting into 
event_resources
- deleting related event rows - cleaning other event rows - updating something that references those rows then lock ordering can conflict and deadlock. Important point about the PostgreSQL message This part:
Copy code
text
Process 120238 waits for ShareLock on transaction ...
blocked by process 120236
does not mean “same query started twice”. It only means: - session/process 
120238
is waiting on a lock held by 
120236
- and 
120236
is also waiting on a lock held by 
120238
That is the circular wait. Those two sessions could be: - vacuum vs vacuum - vacuum vs event insert/persist - vacuum vs other cleanup - vacuum vs API-side event write/read transaction In Prefect, what is the most likely competing process? Most likely one of these: 1. event persister / event writer writing new rows into 
events
and 
event_resources
2. another event cleanup path especially if more than one service loop or replica can trigger cleanup 3. API request handling that emits lifecycle/events newer Prefect versions emit more lifecycle events for domain objects, which increases event table activity Because Prefect uses events quite broadly, event vacuuming can contend with normal live event ingestion. Could background services start two vacuum sessions in parallel? Possibly yes, depending on how the task scheduling/perpetual execution overlaps, especially if: - a prior run hangs long enough - the service loop schedules again - there is no strict singleton protection for that task instance But from your observations, I would be cautious about claiming that without logs proving two simultaneous 
vacuum_events_with_retention_overrides
runs. Given your topology, the safer interpretation is: - one session is vacuum from background services - the other session is likely another concurrent server-side event transaction, potentially from an API pod Why scaling API down to 1 helped That is actually a strong clue. If the deadlock were purely “background pod starts same vacuum twice by itself”, then reducing API replicas from 3 to 1 should matter less. But you observed recovery after reducing everything to: - 1 background - 1 API That suggests cross-replica concurrency from API traffic likely contributed. So my best answer is Yes, two sessions are involved, but they do not have to be two copies of the same background vacuum delete. A vacuum delete can deadlock with another concurrent event-related transaction, and in your environment that is very plausible. How to confirm You’d need PostgreSQL-side logging of deadlock statements or 
pg_stat_activity
around the time of failure to see the exact SQL for both PIDs:
- 
120236
- 
120238
If you can capture 
pg_stat_activity
or deadlock logs from PostgreSQL, that would tell you whether it was: - vacuum vs vacuum - vacuum vs insert - vacuum vs some other event operation If you want, I can help you with a specific Postgres query to capture the blocking sessions next time it happens.
7 Views
Open in Slack
PreviousNext
Powered by