jusstol
08/11/2025, 8:50 AMMarvin
08/11/2025, 8:50 AMMarvin
08/11/2025, 9:01 AMPREFECT_EVENTS_RETENTION_PERIOD
(default is 7 days). Examples: P14D
for 14 days, P30D
for 30 days.
Example (docker-compose/env)
PREFECT_EVENTS_RETENTION_PERIOD=P14D
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
- You can check server services:
prefect server services list-services
Docs:
- Settings reference (see server.events.retention_period)
- Server overview
2) Prune old flow runs so their states and logs go with them
There isn’t a built-in “retention period” for flow runs or logs in Prefect 3. Deleting a flow run will cascade its states and logs, so the safest path is to delete runs older than X days.
Options:
- CLI (one-by-one): you can delete by ID
prefect flow-run delete <FLOW_RUN_ID>
(help: prefect flow-run delete --help
)
- Scripted cleanup via API: enumerate old runs (e.g., older than 90 days and terminal) and delete them. Here’s a minimal Python example using the REST API directly; point PREFECT_API_URL
at your server (e.g., http://localhost:4200/api) and make sure auth is set if needed.
import os
import datetime as dt
import httpx
API = os.environ["PREFECT_API_URL"].rstrip("/")
client = httpx.Client(timeout=30)
# Adjust your cutoff
cutoff = (dt.datetime.utcnow() - dt.timedelta(days=90)).isoformat() + "Z"
# Fetch runs in pages; filter by end_time and terminal state if you like
# Prefect API supports filters in the POST body to /flow_runs/filter or /flow_runs/paginate in 3.x
payload = {
"sort": "START_TIME_ASC",
"flows": {},
"flow_runs": {
"end_time": {"before_": cutoff},
"state": {"type": {"any_": ["COMPLETED","FAILED","CANCELLED","CRASHED"]}}
},
"limit": 200
}
deleted = 0
while True:
r = <http://client.post|client.post>(f"{API}/flow_runs/paginate", json=payload)
r.raise_for_status()
page = r.json()
if not page.get("results"):
break
for fr in page["results"]:
fr_id = fr["id"]
# delete run
d = client.delete(f"{API}/flow_runs/{fr_id}")
# ignore 404s to be resilient
if d.status_code in (200, 204, 404):
deleted += 1
else:
d.raise_for_status()
print(f"Deleted {deleted} flow runs")
Notes:
- Deleting the flow run will also remove its flow_run_state
rows and associated log
rows.
- Test with a very small date window first.
3) If you really want raw SQL (do this carefully)
Back up first. Disable traffic or run during a maintenance window. Then:
- Events (if you need an immediate purge in addition to retention)
-- delete events older than 30 days
DELETE FROM events
WHERE occurred < NOW() - INTERVAL '30 days';
-- clean up any orphaned event_resources (if not cascaded)
DELETE FROM event_resources er
WHERE NOT EXISTS (
SELECT 1 FROM events e WHERE e.id = er.event_id
);
- Flow runs + states + logs (delete runs older than 90 days that are terminal)
If your FK constraints are ON DELETE CASCADE, deleting from flow_run is enough. If you’re unsure, do it in order:
```
-- Identify target runs
WITH old_runs AS (
SELECT id
FROM flow_run
WHERE end_time < NOW() - INTERVAL '90 days'
AND state_type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
)
-- Delete related rows first if cascade is not enabledMarvin
08/11/2025, 9:01 AMThen reclaim space:
VACUUM (VERBOSE, ANALYZE);
-- or to reclaim disk immediately (more intrusive lock):
VACUUM FULL;
```
4) Reduce future growth
- Events: set PREFECT_EVENTS_RETENTION_PERIOD
to a sensible value for your needs.
- Logs volume: reduce what you send to the API (e.g., log at WARN/ERROR for noisy flows, or customize logging to avoid sending every log to Prefect’s logs endpoint). See Logging customization.
- Consider shorter retention in your periodic cleanup script (60–90 days is common).
Quick sanity checklist
- Confirm you’re on Prefect 3.x (so PREFECT_EVENTS_RETENTION_PERIOD
is available).
- Set the retention env var and make sure EventPersister is enabled:
- prefect server services list-services
- Run a small delete test (CLI or script) and verify your UI/API behavior.
- VACUUM afterwards.
If you can share:
- Your desired retention windows (events/logs/runs)
- Whether you prefer API-based cleanup or DB-side SQL
I’ll give you a tailored script and a cron schedule that leaves no trace… eventually.