https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> <@ULVA73B9P> I have regular crashes o...
# ask-marvin
p
@Marvin @Marvin I have regular crashes of my workers with this error, can you tell me with config I can tune to avoid these timeouts ?
Copy code
File "/data/conda/envs/datafetch/lib/python3.12/site-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/conda/envs/datafetch/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/data/conda/envs/datafetch/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout: timed out
06:55:54 PM
prefect.flow_runs
Finished in state Failed('Flow run encountered an exception: ReadTimeout: timed out')
m
✅ thought for 124.3 seconds
Most likely, the main knob to tune is: 
PREFECT_API_REQUEST_TIMEOUT
 That controls the timeout Prefect uses for its API calls, including the read timeout that produces 
httpx.ReadTimeout
. Example:
Copy code
bash
export PREFECT_API_REQUEST_TIMEOUT=120
A few other useful settings: * 
PREFECT_CLIENT_MAX_RETRIES
— how many times Prefect retries transient API failures/timeouts
Copy code
bash
export PREFECT_CLIENT_MAX_RETRIES=7
* 
PREFECT_WORKER_QUERY_SECONDS
— how often a worker polls for work * 
PREFECT_WORKER_HEARTBEAT_SECONDS
— how often a worker heartbeats Increasing these a bit can reduce API chatter if your server/network is under load:
Copy code
bash
export PREFECT_WORKER_QUERY_SECONDS=15
export PREFECT_WORKER_HEARTBEAT_SECONDS=60
* 
PREFECT_FLOWS_HEARTBEAT_FREQUENCY
— flow heartbeat frequency
Copy code
bash
export PREFECT_FLOWS_HEARTBEAT_FREQUENCY=300
A reasonable first attempt would be:
Copy code
bash
export PREFECT_API_REQUEST_TIMEOUT=120
export PREFECT_CLIENT_MAX_RETRIES=7
export PREFECT_WORKER_QUERY_SECONDS=15
export PREFECT_WORKER_HEARTBEAT_SECONDS=60
A couple of important notes: * If this is Prefect internal communication (worker <-> API / flow heartbeat / state updates), these settings apply. * If your own flow code is using 
httpx
, then Prefect settings will not change those requests — you’d need to increase the timeout in your own client code. Given your traceback and the 
prefect.flow_runs
failure, this does look consistent with Prefect API communication timing out. If you're self-hosting Prefect, the root cause can also be server-side slowness (DB/API responsiveness), not just the worker config. In that case, increasing the client timeout helps, but you may also want to inspect server performance. Relevant references: * Prefect docs * Prefect source If you want, I can help you map where to set these vars based on how your worker is running (systemd, Docker, Kubernetes, ECS, etc.).
Your message was too long, here's your message at the allowed limit: here is the diagnotic of Claude do you agree ? Can you explain me the diffrent variables he propose to tune ? ``` The Prefect event_persister can't keep up with DELETE on its own tables Every 15 minutes it runs a trim to delete events older than 1 day; every single attempt in this log times out at 30 s (your PREFECT_SERVER_DATABASE_TIMEOUT) 03 52 47 Error trimming events and resources asyncpg/protocol/protocol pyx 206 in bind_execute TimeoutError 04 08 17 Error trimming… (15 min later — same) 04 23 48 Error trimming… 04 39 53 Error trimming… 17 in a row never a success The trim is DELETE FROM events WHERE occurred < now() - interval (event_persister py 184) Because it always times out events keep accumulating and the table keeps growing → every subsequent trim is slower → also the queries used by set_state and /labels that scan related rows start exceeding the same 30 s timeout That's where your worker‑visible 500s and httpx ReadTimeout come from They cluster right after a failed trim cycle — e g trim fails at 03 52 47 first 500 hits at 04 02 06 Supporting evidence - Only 8 of 34 server 500s are connect-timeouts (fresh asyncpg) The other 26 are statement timeouts on existing pooled connections — i e PG is slow not unreachable - 931 ws //…/events/in handshake timeouts The websocket lives in the same uvicorn process; when the event_persister holds the loop the WS endpoint can't even handshake - Loop overruns line up RecentDeploymentsScheduler took 189 s happens at 17 33 and a trim fails at 17 29 56 — the long trim ran from ~17 30 to 17 30+30 s leaving the event loop unable to service other coroutines - Concurrency is moderate (~10 flows starting per minute; ~21 in the worst minute at HH 59 from late-run backlog) not a thundering-herd What to do 1 Catch up the events and event_resources tables manually then keep them small -- one-shot while server is running DELETE FROM events WHERE occurred < now() - interval '6 hours'; DELETE FROM event_resources WHERE updated < now() - interval '6 hours'; VACUUM (VERBOSE ANALYZE) events; VACUUM (VERBOSE ANALYZE) event_resources; 1 Run the DELETE in small chunks (LIMIT 50000 in a loop) so it doesn't lock anything for long Once they're small regular trim will keep up 2 Verify the index used by the trim \d events EXPLAIN DELETE FROM events WHERE occurred < now() - interval '1 day'; 2 It should hit ix_events__occurred If it's doing a Seq Scan that's the entire problem 3 Temporarily lower retention so trims can succeed while you recover PREFECT_EVENTS_RETENTION_PERIOD 6 hours 4 Raise the DB statement timeout to a value the trim can actually hit e g PREFECT_SERVER_DATABASE_TIMEOUT 120 4 30 s is what's actually killing the trim today With 120 s the trim has a chance to complete even when the table is fat 5 Bump the server pool a little so background services + uvicorn requests
p
@Marvin here is the diagnotic of Claude, do you agree ? Can you explain me the diffrent variables he propose to tune ? The Prefect event_persister can't keep up with DELETE on its own tables. Every 15 minutes it runs a trim to delete events older than 1 day; every single attempt in this log times out at 30 s (your PREFECT_SERVER_DATABASE_TIMEOUT): 035247 Error trimming events and resources asyncpg/protocol/protocol.pyx:206, in bind_execute TimeoutError 040817 Error trimming… (15 min later — same) ... The trim is DELETE FROM events WHERE occurred < now() - interval (event_persister.py:184). Because it always times out, events keep accumulating, and the table keeps growing → every subsequent trim is slower → also the queries used by set_state and /labels that scan related rows start exceeding the same 30 s timeout. That's where your worker‑visible 500s and httpx.ReadTimeout come from. They cluster right after a failed trim cycle — e.g. trim fails at 035247, first 500 hits at 040206. 3. Temporarily lower retention so trims can succeed while you recover: PREFECT_EVENTS_RETENTION_PERIOD: "6 hours" 4. Raise the DB statement timeout to a value the trim can actually hit, e.g.: PREFECT_SERVER_DATABASE_TIMEOUT: "120" 4. 30 s is what's actually killing the trim today. With 120 s the trim has a chance to complete even when the table is fat. 5. Bump the server pool a little so background services + uvicorn requests don't fight for the same 25 slots while a slow trim hogs one: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE: "20" PREFECT_SQLALCHEMY_MAX_OVERFLOW: "30"
m
✅ thought for 148.7 seconds
Yes — the diagnosis is plausible in general, but I don’t fully agree with the exact details as written without checking your Prefect version. The biggest thing I’d correct is this: In current Prefect 3 code, event cleanup is handled by a DB vacuum service, not by the event persister itself. So the shape of the diagnosis makes sense: - event/event-resource cleanup is timing out - old rows accumulate - DB gets slower - API requests can start failing or timing out - workers then see 
httpx.ReadTimeout
/ failed state updates But I would treat the specific implementation details (
every 15 minutes
, exact DELETE statement, exact service name) as version-dependent. What I do agree with: - if your event tables are growing and cleanup cannot complete inside the DB statement timeout, - then raising DB timeout and/or reducing retained events can absolutely help, - and DB pool sizing can also matter if background cleanup and API traffic compete for connections. Here’s what the proposed variables mean. --- ## 1. 
PREFECT_EVENTS_RETENTION_PERIOD
This controls how long Prefect keeps events before cleanup removes them. If you lower it, Prefect has fewer old rows to retain, so cleanup work gets easier. Example:
Copy code
bash
PREFECT_EVENTS_RETENTION_PERIOD="6 hours"
What it does: - keeps less historical event data - reduces event table growth - can make cleanup faster and more likely to finish Tradeoff: - you lose older event history sooner Good use: - temporary recovery when cleanup is badly behind - permanently, if you don’t need long event retention --- ## 2. 
PREFECT_SERVER_DATABASE_TIMEOUT
This is one of the most important settings in your case. It controls the DB statement timeout for Prefect server queries. If a cleanup 
DELETE
or some API query takes longer than this, PostgreSQL cancels it. Example:
Copy code
bash
PREFECT_SERVER_DATABASE_TIMEOUT="120"
What it does: - allows slow queries more time to complete - especially useful for large cleanup deletes or expensive reads Why it matters: - if cleanup always dies at 30s, it never catches up - if it never catches up, table growth can snowball - then other queries can also get slower Tradeoff: - very slow queries are allowed to run longer - this can hide an underlying indexing/table-growth problem if used alone My view: - yes, increasing this is a very reasonable mitigation - but it’s best paired with retention reduction and general DB health checks --- ## 3. 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
This sets the base number of DB connections Prefect server keeps in its SQLAlchemy pool. Example:
Copy code
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE="20"
What it does: - more concurrent server work can access the DB without waiting - background services and API requests are less likely to contend for a small pool Why it may help: - if cleanup jobs, API handlers, and background services are all active, a tiny pool can become a bottleneck Tradeoff: - more DB connections used overall - if PostgreSQL is already resource-constrained, blindly increasing this can make things worse Important: - this only helps if connection contention is actually part of the problem - it does not fix a fundamentally slow delete/query by itself --- ## 4. 
PREFECT_SQLALCHEMY_MAX_OVERFLOW
This controls how many extra temporary connections can be opened beyond the base pool size. So if: - 
POOL_SIZE=20
- 
MAX_OVERFLOW=30
then Prefect can temporarily use up to 50 connections under load. Example:
Copy code
bash
PREFECT_SQLALCHEMY_MAX_OVERFLOW="30"
What it does: - gives the server burst capacity during spikes - reduces request failures due to pool exhaustion Tradeoff: - more possible pressure on Postgres - if your DB max connections are low, you can hit DB-side limits One note: - depending on Prefect version, there may also be a server-prefixed variant for this setting - if you want, I can help verify the exact env var names for your installed version
--- ## Do I think these are good knobs to try? Yes, broadly: ### Most likely useful first 1. Increase 
PREFECT_SERVER_DATABASE_TIMEOUT
2. Decrease 
PREFECT_EVENTS_RETENTION_PERIOD
These directly address “cleanup cannot finish”. ### Possibly useful if DB/server is under concurrency pressure 3. Increase 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
4. Increase 
PREFECT_SQLALCHEMY_MAX_OVERFLOW
These help more with contention than with a single slow query. --- ## Suggested cautious tuning Instead of jumping immediately to very large values, I’d try something like:
Copy code
bash
PREFECT_EVENTS_RETENTION_PERIOD="12 hours"
PREFECT_SERVER_DATABASE_TIMEOUT="120"
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE="10"
PREFECT_SQLALCHEMY_MAX_OVERFLOW="20"
Then watch: - server logs for cleanup success/failure - event table growth - API latency - worker 
ReadTimeout
frequency --- ## One more important distinction These are server-side settings. Earlier, I mentioned: - 
PREFECT_API_REQUEST_TIMEOUT
- 
PREFECT_CLIENT_MAX_RETRIES
Those are still useful too, but they are client/worker-side mitigations. So the two layers are: ### Server side Fix the cause: - 
PREFECT_EVENTS_RETENTION_PERIOD
- 
PREFECT_SERVER_DATABASE_TIMEOUT
- DB pool sizing ### Worker/client side Make workers more tolerant while server is slow: - 
PREFECT_API_REQUEST_TIMEOUT
- 
PREFECT_CLIENT_MAX_RETRIES
If your server is timing out internally, I’d prioritize the server-side fixes first. --- ## My short verdict Yes, I agree with the overall diagnosis. Especially: - cleanup timing out - event data accumulating - DB slowness propagating into API timeouts seen by workers But I’d phrase it as: - “very likely” rather than - “definitively proven” unless you also have: - growing event table size - repeated cleanup timeout logs - API latency spikes at the same times If you want, I can help you build a step-by-step recovery plan for your self-hosted Prefect server, including which settings to change first and how to validate that the cleanup is actually catching up.
✅ thought for 22.1 seconds
p
@Marvin I'm using prefect version 3.5.0
m
✅ thought for 75.5 seconds
Got it — for Prefect 3, I’d say: Yes, the diagnosis is directionally correct, and the knobs Claude suggested are the right kind of server-side settings to inspect for a self-hosted deployment. But there’s one important refinement: In Prefect 3, event cleanup is handled by the DB vacuum / cleanup service, not “the event persister can’t keep up” in the old sense. So I’d restate the diagnosis like this: - Prefect’s event/event-related cleanup queries are timing out - old rows accumulate - the database gets slower - API requests may then slow down or fail - workers/flow runs see 
httpx.ReadTimeout
when talking to the API That part is very believable. ## What each variable does ### 
PREFECT_EVENTS_RETENTION_PERIOD
How long Prefect keeps events before they are eligible for deletion. Example:
Copy code
bash
PREFECT_EVENTS_RETENTION_PERIOD="6 hours"
Effect: - lower value = fewer old events kept - cleanup has less data to manage - useful when event tables have grown too large Tradeoff: - shorter observability/history window --- ### 
PREFECT_SERVER_DATABASE_TIMEOUT
How long Prefect allows a DB statement to run before cancelling it. Example:
Copy code
bash
PREFECT_SERVER_DATABASE_TIMEOUT="120"
Effect: - lets slow cleanup queries and slow API-related queries finish - especially relevant if you currently see repeated timeout errors at 30s Tradeoff: - slow queries now run longer, so this helps symptoms but may not fully solve root DB performance issues This is probably the highest-value setting to try first on the server side. --- ### 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
The baseline number of DB connections in Prefect server’s SQLAlchemy connection pool. Example:
Copy code
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE="20"
Effect: - more simultaneous DB work can proceed without waiting for a free connection - helps if API requests and background cleanup jobs are competing Tradeoff: - uses more Postgres connections --- ### 
PREFECT_SQLALCHEMY_MAX_OVERFLOW
How many extra DB connections can be created above the pool size during bursts. Example:
Copy code
bash
PREFECT_SQLALCHEMY_MAX_OVERFLOW="30"
If pool size is 20 and overflow is 30, Prefect may temporarily use up to 50 DB connections. Effect: - helps with bursts of API traffic or background work - reduces connection starvation Tradeoff: - can put more pressure on Postgres if the DB is already tight --- ## Do I agree with the proposed changes? ### Yes, especially these two:
Copy code
bash
PREFECT_EVENTS_RETENTION_PERIOD="6 hours"
PREFECT_SERVER_DATABASE_TIMEOUT="120"
Those directly address: - too much event history - cleanup queries being killed before they finish ### Maybe, depending on your DB capacity:
Copy code
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE="20"
PREFECT_SQLALCHEMY_MAX_OVERFLOW="30"
These are good if connection contention is part of the problem. If your Postgres instance is small, jumping straight to 20/30 may be too aggressive. A more conservative first step might be:
Copy code
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE="10"
PREFECT_SQLALCHEMY_MAX_OVERFLOW="20"
--- ## My recommended order For Prefect 3, I’d try this order: ### 1. Server-side first
Copy code
bash
PREFECT_EVENTS_RETENTION_PERIOD="6 hours"
PREFECT_SERVER_DATABASE_TIMEOUT="120"
### 2. Then, if needed, increase pool capacity
Copy code
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE="10"
PREFECT_SQLALCHEMY_MAX_OVERFLOW="20"
### 3. Also make workers more tolerant On the worker side:
Copy code
bash
PREFECT_API_REQUEST_TIMEOUT="120"
PREFECT_CLIENT_MAX_RETRIES="7"
That won’t fix the DB issue, but it can reduce worker crashes while the server is under pressure. --- ## Practical interpretation - Server settings help fix the cause - Worker/client settings help reduce the symptoms Since you’re on Prefect 3 and this sounds self-hosted, I’d focus on server DB cleanup/timeout settings first.
If you want, I can help you turn this into an exact config snippet for: - Docker Compose - Kubernetes - Helm values - systemd / bare-metal env vars
Yes — for Prefect 3.5.0, Claude’s diagnosis is much closer to exact, and I agree with it. In 3.5.0 specifically: - event cleanup is handled in the event persister - cleanup runs every 15 minutes - retention defaults to 7 days - DB statement timeout is a key limiter - connection pool settings can also matter So thanks for giving the version — that changes the answer. ## What this means in your case If your logs show repeated failures trimming old events/resources, then this can absolutely lead to: - event tables growing continuously - trim queries getting slower over time - other API queries slowing down too - worker-visible failures like `500`s or 
httpx.ReadTimeout
So yes, the proposed tuning makes sense for a self-hosted Prefect 3.5.0 server. --- ## The variables, explained ### 
PREFECT_EVENTS_RETENTION_PERIOD
How long Prefect keeps events before deleting them. Default in 3.5.0: - 7 days What lowering it does: - reduces how much old event data must be retained - makes cleanup easier/faster - helps the system catch up if it is behind Example:
Copy code
bash
PREFECT_EVENTS_RETENTION_PERIOD="6 hours"
When to use it: - if you do not need long event history - as a temporary recovery measure when tables have grown too much Tradeoff: - you lose older event history sooner Also valid alias:
Copy code
bash
PREFECT_SERVER_EVENTS_RETENTION_PERIOD
--- ### 
PREFECT_SERVER_DATABASE_TIMEOUT
This is the statement timeout for Prefect server DB operations. Default in 3.5.0: - 10 seconds If your trim/delete query takes longer than this, Postgres cancels it. Example:
Copy code
bash
PREFECT_SERVER_DATABASE_TIMEOUT="120"
What increasing it does: - gives cleanup queries more time to complete - also gives other slower DB queries more time Why it matters: - if cleanup always dies at the timeout, it never catches up - if it never catches up, DB size and query cost keep increasing Tradeoff: - slow queries can run longer before failing - if the root cause is severe DB slowness, this is mitigation, not a full fix Also valid alias:
Copy code
bash
PREFECT_API_DATABASE_TIMEOUT
--- ### 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
Base number of persistent DB connections Prefect server keeps. Default in 3.5.0: - 5 Example:
Copy code
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE="20"
What increasing it does: - allows more concurrent DB work - helps if API handlers and background services are competing for too few connections Why it may help: - event persister cleanup, API requests, and other server jobs can all contend for DB connections Tradeoff: - increases load on PostgreSQL - should stay consistent with your Postgres max connections and available resources Alias also works:
Copy code
bash
PREFECT_SQLALCHEMY_POOL_SIZE
--- ### 
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
Extra temporary DB connections allowed above 
POOL_SIZE
. Default in 3.5.0: - 10 Example:
Copy code
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW="30"
If: - pool size = 20 - max overflow = 30 then Prefect can use up to 50 DB connections during bursts. What it does: - gives burst capacity under load - can reduce waiting/failures when many requests hit at once Tradeoff: - more possible stress on Postgres - should be tuned with DB limits in mind Alias also works:
Copy code
bash
PREFECT_SQLALCHEMY_MAX_OVERFLOW
--- ## One more useful variable for 3.5.0 ### 
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE_DELETE
This one is especially relevant in your situation. Default in 3.5.0: - 10000 It controls how many rows are deleted per cleanup batch. Example:
Copy code
bash
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE_DELETE="2000"
Why it matters: - if each delete batch is too large, it may exceed the DB timeout - lowering the batch size can make each delete complete faster - this can be better than only increasing timeout Tradeoff: - smaller batches may require more cycles to fully catch up
This is an important knob because it targets the cleanup job directly. --- ## Do I agree with the proposed settings? ### Good ideas I agree these are sensible knobs: - 
PREFECT_EVENTS_RETENTION_PERIOD
- 
PREFECT_SERVER_DATABASE_TIMEOUT
- 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
- 
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
### But I would add / emphasize Also consider: - 
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE_DELETE
Because if cleanup times out, changing the delete batch size is often a very practical lever. --- ## What I would try first For a safe first pass, I’d do something like:
Copy code
bash
PREFECT_EVENTS_RETENTION_PERIOD="12 hours"
PREFECT_SERVER_DATABASE_TIMEOUT="60"
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE="10"
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW="20"
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE_DELETE="2000"
Why this set: - lower retention reduces backlog - higher timeout gives deletes a chance to finish - smaller delete batches reduce per-query runtime - a somewhat larger pool reduces contention If your DB is healthy and the main issue is just oversized cleanup work, this is a reasonable recovery configuration. --- ## What I would not do first I would avoid jumping immediately to: - very large pool sizes like 20/30 unless you know Postgres can handle it - extremely long DB timeout values without also reducing delete batch size or retention Because: - bigger pools can overload the DB - longer timeout alone can just mean “slow queries hang around longer” --- ## Suggested order of operations 1. Reduce retention
Copy code
bash
PREFECT_EVENTS_RETENTION_PERIOD="6 hours"
2. Increase DB timeout moderately
Copy code
bash
PREFECT_SERVER_DATABASE_TIMEOUT="60"
3. Reduce delete batch size
Copy code
bash
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE_DELETE="2000"
4. Increase pool size only if needed
Copy code
bash
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE="10"
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW="20"
5. Watch server logs to confirm trims start succeeding --- ## Short answer For Prefect 3.5.0: yes, I agree with Claude’s diagnosis overall. And the key variables mean: - 
PREFECT_EVENTS_RETENTION_PERIOD
 → keep fewer old events - 
PREFECT_SERVER_DATABASE_TIMEOUT
 → allow DB queries/deletes more time - 
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
 → more base DB connections - 
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
 → more burst DB connections - 
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE_DELETE
 → smaller/larger delete batches for cleanup If you want, I can turn this into a recommended Helm/Docker/Kubernetes env block for Prefect 3.5.0.
p
@Marvin in fact I don't really care about events, I just need them to trigger some automations but I never consult events through the UI, so maybe they should not be persistent ?
m
✅ thought for 17.2 seconds
Yes — if you only need events to drive automations and you do not need to browse historical events/resources in the UI, then reducing or even disabling event persistence may be a good fit. But there’s an important distinction: - Automations need events to exist long enough to be processed - Persistence is what stores them in the database for later querying/history So the question is not “do I need events at all?” — you do, because automations depend on them. The question is whether you need them stored durably for history/search/UI use. ## For Prefect 3.5.0 The event persister service is what writes events to the DB, and it can be disabled. The relevant setting is:
Copy code
bash
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED="false"
## What happens if you disable it Likely benefits: - no event table growth - no trim/delete pressure - much less DB load from event persistence Likely tradeoff: - you lose persisted event history - event/resource views in the UI may be empty or incomplete - anything relying on querying historical events from the DB will no longer work as expected What should still work: - real-time event handling for automations, assuming your automations only depend on the event stream and not on historical persisted event queries ## My recommendation based on your use case If your real requirement is: - “automations must still trigger” - “I don’t care about browsing past events in the UI” then this is a very reasonable option to test:
Copy code
bash
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED="false"
I would test it carefully in a non-prod or controlled environment first and verify: 1. automations still trigger correctly 2. UI event history disappears or becomes limited, which you’re okay with 3. DB pressure drops 4. worker/API timeouts improve ## Safer middle ground If you’re not ready to fully disable persistence, a lower-risk option is:
Copy code
bash
PREFECT_EVENTS_RETENTION_PERIOD="1 hour"
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE_DELETE="2000"
PREFECT_SERVER_DATABASE_TIMEOUT="60"
That keeps events only briefly: - enough for short-term needs - much less DB buildup - lower cleanup cost This is often a good compromise if you’re unsure whether some hidden feature depends on persisted events. ## My opinion Given what you said, I’d consider these two options: ### Option A — most aggressive Disable persistence entirely:
Copy code
bash
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED="false"
### Option B — conservative Keep persistence but make it minimal:
Copy code
bash
PREFECT_EVENTS_RETENTION_PERIOD="1 hour"
PREFECT_SERVER_DATABASE_TIMEOUT="60"
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE_DELETE="2000"
If your priority is stability over event history, Option A is attractive. If you want, I can help you confirm from Prefect 3.5.0 behavior whether automations continue to work with 
EVENT_PERSISTER_ENABLED=false
, because that’s the one detail I’d want to verify before recommending it as a production change.
p
@Marvin it will not impact the manual restart of some failed tasks, or the internal retries in the flow ?
m
✅ thought for 142.7 seconds
Short answer: no, it should not affect either of those. In Prefect 3.5.0: - manual restart/rerun relies on run state orchestration, not persisted events - built-in task/flow retries also rely on state transitions and orchestration rules, not persisted events So disabling event persistence should not break: - rerunning/restarting failed flow runs - task retries - flow retries - normal state transitions/history for runs ## The key distinction Prefect has two related but separate concepts: ### 1. States — core orchestration These are the important records for: - flow/task run lifecycle - retries - restarts/reruns - failed/completed/running state history These are stored in the run/state tables and are the source of truth. ### 2. Events — observability/automation stream These are mainly for: - automations - event-driven reactions - event history in the UI - event/resource-related observability So if you disable event persistence, you are not disabling state persistence. That’s why: - retries still work - reruns still work - orchestration still works ## What disabling event persistence would affect If you set:
Copy code
bash
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED="false"
you should expect impact mainly around: - historical event browsing - event/resource views in the UI - anything that depends on querying old persisted events later ## What should still work These should still be fine: - manual rerun/restart of flow runs - task retries defined with 
retries=
 - flow retries defined with 
retries=
 - regular orchestration state updates - real-time automations that consume live events ## Small caution I’d still phrase one thing carefully: - reactive / real-time automations should continue to work - if you have any feature or custom process that expects to query old events from the DB later, that would be affected But based on your described use case — “I just need automations, I never consult events in the UI” — disabling persistence sounds compatible. ## Practical answer If your question is:
Will disabling event persistence impact manual restart of failed tasks, or internal retries in the flow?
My answer is: No, not in normal Prefect 3.5.0 operation. Those depend on state orchestration, not persisted events. If you want, I can help you with a safe rollout plan: 1. disable event persistence 2. verify automations still fire 3. verify rerun/retry behavior 4. confirm DB pressure drops
2 Views
Open in Slack
PreviousNext
Powered by