Rasmus Simmelkjær Berntsen
11/10/2025, 7:15 AMMarvin
11/10/2025, 7:15 AMMarvin
11/10/2025, 7:33 AM# prefect.toml on the server
[server.events]
retention_period = "P14D" # ISO-8601 duration, e.g. P7D, P14D, P30D
- Env var equivalents: PREFECT_SERVER_EVENTS_RETENTION_PERIOD (and PREFECT_EVENTS_RETENTION_PERIOD).
- Docs: Settings reference
2) Database-side cleanup (recommended for volume)
Run these Postgres examples as a scheduled job (cron, Kubernetes CronJob, etc.). Always preview with SELECT first and run in batches to avoid long locks.
- Delete old logs
-- Preview
SELECT count(*) FROM log WHERE timestamp < NOW() - INTERVAL '30 days';
-- Delete in batches
DELETE FROM log
WHERE id IN (
SELECT id FROM log
WHERE timestamp < NOW() - INTERVAL '30 days'
LIMIT 10000
);
- Delete old events
-- Preview
SELECT count(*) FROM events WHERE occurred < NOW() - INTERVAL '14 days';
-- Delete in batches
DELETE FROM events
WHERE id IN (
SELECT id FROM events
WHERE occurred < NOW() - INTERVAL '14 days'
LIMIT 10000
);
- Delete old run history
It’s safest to delete state rows before runs if your DB doesn’t have cascading FKs. Use start_time (execution start), not created time.
-- Flow run states older than 90 days
DELETE FROM flow_run_state frs
USING flow_run fr
WHERE frs.flow_run_id = fr.id
AND fr.start_time < NOW() - INTERVAL '90 days';
-- Task run states older than 90 days
DELETE FROM task_run_state trs
USING task_run tr
WHERE trs.task_run_id = tr.id
AND tr.start_time < NOW() - INTERVAL '90 days';
-- Now delete the runs themselves
DELETE FROM task_run WHERE start_time < NOW() - INTERVAL '90 days';
DELETE FROM flow_run WHERE start_time < NOW() - INTERVAL '90 days';
Post‑cleanup: run VACUUM (ANALYZE) during a maintenance window; for big tables, consider pg_repack instead of VACUUM FULL.
Reference: Database maintenance guide
3) Orchestrate cleanup with a small Prefect job
If you prefer Python, you can schedule a Prefect flow that runs the SQL above (e.g., using psycopg/asyncpg) on a cadence. For very large datasets, stick with server-side SQL in batches as shown.
Notes and limits
- There is no public REST endpoint to bulk delete logs by time window; logs/events cleanup is typically done with SQL. Details: Logs API (read-only for time filters).
- Per-run deletions via API are available if you want to remove specific runs:
- Delete a single flow run: DELETE /flow-runs/{id}
- If you use API for runs, you may still need SQL to remove logs/events at scale.
A few best practices
- Run deletes in chunks (e.g., 10k rows) and during off-peak hours.
- Monitor table sizes via pg_stat tables; consider partitioning events/logs if very large.
- Test queries with SELECT first; validate table names in your DB (\dt) since schemas can change between versions.
If you can share:
- Your DB backend (Postgres version),
- Desired retention windows (events/logs/runs), and
- Your deployment style (Docker, Helm, bare metal)
…I can tailor exact SQL (including batch sizes) and a ready-to-run cleanup job for your environment.