<@ULVA73B9P> How can i configure a self hosted pre...
# ask-marvin
r
@Marvin How can i configure a self hosted prefect server to delete things like old logs, events etc from the databae to avoid it filling up?
m
thought for 1070.4 seconds
Short answer: there isn’t an automatic “retention/cleanup” job built into a self‑hosted Prefect 3 server. You’ll want to set a retention window for events and then run periodic cleanup (SQL or a small Prefect flow/cron) to prune old events, logs, and historical runs. Here’s a practical setup you can use. 1) Set an events retention window (advisory) - Prefect exposes a setting for how long events are considered “retained” but it does not automatically prune your DB. - Configure via environment variable or prefect.toml:
Copy code
# prefect.toml on the server
[server.events]
retention_period = "P14D"  # ISO-8601 duration, e.g. P7D, P14D, P30D
- Env var equivalents:
PREFECT_SERVER_EVENTS_RETENTION_PERIOD
(and
PREFECT_EVENTS_RETENTION_PERIOD
). - Docs: Settings reference 2) Database-side cleanup (recommended for volume) Run these Postgres examples as a scheduled job (cron, Kubernetes CronJob, etc.). Always preview with SELECT first and run in batches to avoid long locks. - Delete old logs
Copy code
-- Preview
SELECT count(*) FROM log WHERE timestamp < NOW() - INTERVAL '30 days';

-- Delete in batches
DELETE FROM log
WHERE id IN (
  SELECT id FROM log
  WHERE timestamp < NOW() - INTERVAL '30 days'
  LIMIT 10000
);
- Delete old events
Copy code
-- Preview
SELECT count(*) FROM events WHERE occurred < NOW() - INTERVAL '14 days';

-- Delete in batches
DELETE FROM events
WHERE id IN (
  SELECT id FROM events
  WHERE occurred < NOW() - INTERVAL '14 days'
  LIMIT 10000
);
- Delete old run history It’s safest to delete state rows before runs if your DB doesn’t have cascading FKs. Use start_time (execution start), not created time.
Copy code
-- Flow run states older than 90 days
DELETE FROM flow_run_state frs
USING flow_run fr
WHERE frs.flow_run_id = fr.id
  AND fr.start_time < NOW() - INTERVAL '90 days';

-- Task run states older than 90 days
DELETE FROM task_run_state trs
USING task_run tr
WHERE trs.task_run_id = tr.id
  AND tr.start_time < NOW() - INTERVAL '90 days';

-- Now delete the runs themselves
DELETE FROM task_run WHERE start_time < NOW() - INTERVAL '90 days';
DELETE FROM flow_run WHERE start_time < NOW() - INTERVAL '90 days';
Post‑cleanup: run
VACUUM (ANALYZE)
during a maintenance window; for big tables, consider
pg_repack
instead of
VACUUM FULL
. Reference: Database maintenance guide 3) Orchestrate cleanup with a small Prefect job If you prefer Python, you can schedule a Prefect flow that runs the SQL above (e.g., using psycopg/asyncpg) on a cadence. For very large datasets, stick with server-side SQL in batches as shown. Notes and limits - There is no public REST endpoint to bulk delete logs by time window; logs/events cleanup is typically done with SQL. Details: Logs API (read-only for time filters). - Per-run deletions via API are available if you want to remove specific runs: - Delete a single flow run: DELETE /flow-runs/{id} - If you use API for runs, you may still need SQL to remove logs/events at scale. A few best practices - Run deletes in chunks (e.g., 10k rows) and during off-peak hours. - Monitor table sizes via pg_stat tables; consider partitioning events/logs if very large. - Test queries with SELECT first; validate table names in your DB (
\dt
) since schemas can change between versions. If you can share: - Your DB backend (Postgres version), - Desired retention windows (events/logs/runs), and - Your deployment style (Docker, Helm, bare metal) …I can tailor exact SQL (including batch sizes) and a ready-to-run cleanup job for your environment.