<@ULVA73B9P> What are the default data retention s...
# ask-marvin
b
@Marvin What are the default data retention settings for things like task runs, logs, states, event, etc for open source self-hosted Prefect 3? The internal Postgres database volume is around 33GB and I would like to limit how much it grow by applying some retention policies
m
thought for 252.1 seconds
Short answer: In self‑hosted Prefect 3, only Events have a default retention policy. Everything else is kept indefinitely unless you enable the db vacuum for flow runs. Here are the exact defaults and how to change them: - Events - Default retention: 7 days - Auto-cleanup: Enabled by default (when event persister is enabled, which it is by default) - Config:
PREFECT_SERVER_EVENTS_RETENTION_PERIOD
(alias:
PREFECT_EVENTS_RETENTION_PERIOD
) - Notes: You can override by event type (e.g., heartbeats) with
PREFECT_SERVER_SERVICES_DB_VACUUM_EVENT_RETENTION_OVERRIDES
- Flow runs - Default retention: None (no cleanup unless enabled) - Auto-cleanup: Disabled by default - If you enable cleanup, default retention is 90 days (based on end_time) and applies only to top‑level runs; subflow runs are not deleted by the vacuum - Config to enable:
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED="events,flow_runs"
- Retention window:
PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD
(e.g.,
14d
,
P30D
) - Task runs - Default retention: None on their own - Deletion behavior: Removed via database cascade only when their parent flow run is deleted - States (flow run and task run states) - Default retention: None on their own - Deletion behavior: Removed via cascade when their parent run is deleted - Logs - Default retention: None on their own - Deletion behavior: Removed when parent flow run is deleted; orphaned logs are also pruned by the vacuum - Artifacts and artifact collections - Default retention: None on their own - Deletion behavior: Removed when parent flow run is deleted; the vacuum also prunes orphaned artifacts and reconciles collections - Subflow runs - Not deleted by the flow run vacuum (it only considers top‑level runs) - If a parent task run is deleted, the subflow run remains and its parent reference is cleared (FK is ON DELETE SET NULL) Other vacuum service defaults (when you enable it for flow runs): - Loop interval:
PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS=3600
(1 hour) - Batch size:
PREFECT_SERVER_SERVICES_DB_VACUUM_BATCH_SIZE=200
Example: keep events 3 days, flow runs 30 days, run every 30 mins
Copy code
# Enable cleanup for both events and flow runs
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED="events,flow_runs"

# Event retention (global) – 3 days
PREFECT_SERVER_EVENTS_RETENTION_PERIOD="3d"

# Optional: event-specific overrides (e.g., heartbeats)
PREFECT_SERVER_SERVICES_DB_VACUUM_EVENT_RETENTION_OVERRIDES='{"prefect.flow-run.heartbeat": "P2D"}'

# Flow run retention – 30 days
PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD="30d"

# Service cadence and batch sizing
PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS="1800"
PREFECT_SERVER_SERVICES_DB_VACUUM_BATCH_SIZE="500"
Notes and gotchas - Event vacuum runs only if events are being persisted (default) and
"events"
is enabled in
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED
(default). - Flow run vacuum will only delete terminal, top‑level runs older than the retention period; it won’t chase into subflows. - There is no dedicated CLI command to prune data. Use environment variables/settings as above and restart the server, or write a small Prefect flow/script using the API to delete old runs if you want finer control. Docs and source for deeper reference - Database maintenance guide: Database maintenance (Prefect 3) - Vacuum service API reference: DB vacuum service API - Server services settings: Server services settings - Source (vacuum): db_vacuum.py - Source (settings): server services settings
- Source (events settings): events settings - Schema note: subflow FK is ON DELETE SET NULL (so subflows persist if parent task run is deleted): ORM models If you share your target retention (e.g., “30 days for runs, 3 days for events”) and how you deploy Prefect (Docker Compose, Helm, etc.), I can suggest an exact env/config snippet to keep that 33GB volume in check.