Christian Dalsvaag
11/27/2025, 11:17 AMMarvin
11/27/2025, 11:17 AMMarvin
11/27/2025, 11:24 AMprefect.db and trim it.
Assuming Prefect 3.x with the default local server and SQLite:
- The biggest space users are usually:
- events (high-volume; default retention 7 days)
- logs (all flow/task logs sent to the API; no default retention)
- run state history (flow_run_state/task_run_state)
First, let’s confirm what’s actually large in your DB
Run these against your SQLite DB (typically at ~/.prefect/prefect.db):
-- Row counts for key tables
SELECT 'events' AS t, COUNT(*) FROM events
UNION ALL SELECT 'log', COUNT(*) FROM log
UNION ALL SELECT 'flow_run_state', COUNT(*) FROM flow_run_state
UNION ALL SELECT 'task_run_state', COUNT(*) FROM task_run_state
UNION ALL SELECT 'flow_run', COUNT(*) FROM flow_run
UNION ALL SELECT 'artifact', COUNT(*) FROM artifact
ORDER BY t;
-- Approx content size by summing text/json column lengths
SELECT 'log_bytes' AS what, IFNULL(SUM(LENGTH(message)),0) FROM log
UNION ALL SELECT 'events_bytes', IFNULL(SUM(
LENGTH(event) + COALESCE(LENGTH(resource),0) + COALESCE(LENGTH(related),0) + COALESCE(LENGTH(payload),0)
),0) FROM events;
-- How far back your data goes (helps confirm retention)
SELECT 'events' AS t, MIN(occurred), MAX(occurred) FROM events
UNION ALL SELECT 'log', MIN(timestamp), MAX(timestamp) FROM log
UNION ALL SELECT 'flow_run', MIN(created), MAX(created) FROM flow_run;
-- DB stats
PRAGMA page_count;
PRAGMA page_size;
PRAGMA freelist_count;
PRAGMA integrity_check;
Common findings you may see:
- Lots of events for relatively few runs (each state change emits events)
- Many logs if your flows default to INFO logging and send logs to API
- SQLite file size won’t shrink after deletes until you run VACUUM
Quick ways to reduce growth going forward
- Lower event retention (server setting; default ~7 days)
- Use a tighter window like 2–3 days if you don’t need long-lived event history:
prefect config set PREFECT_API_EVENTS_RETENTION_PERIOD="2d"
Restart the API; events older than the retention will be batch-trimmed automatically.
- Reduce or disable logging to the API (client setting)
- If you do not need logs in the UI/API:
prefect config set PREFECT_LOGGING_TO_API_ENABLED=false
- Or at least reduce verbosity:
prefect config set PREFECT_LOGGING_LEVEL="WARNING"
This dramatically cuts DB writes to the log table.
- Optional server-side tuning for event cleanup throughput (helps large backlogs clear faster):
prefect config set PREFECT_API_SERVICES_EVENT_PERSISTER_BATCH_SIZE="50"
prefect config set PREFECT_API_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL="3"
Then restart the API.
Clean up existing data now
- Delete old flow runs (and their states, artifacts, logs) programmatically:
```
import asyncio
from datetime import datetime, timedelta, timezone
from prefect.client.orchestration import get_client
from prefect.client.schemas.filters import FlowRunFilter, FlowRunFilterStartTime, FlowRunFilterState, FlowRunFilterStateType
from prefect.states import StateType
async def delete_old_flow_runs(days_to_keep=14, batch_size=100):
cutoff = datetime.now(timezone.utc) - timedelta(days=days_to_keep)
async with get_client() as client:
fr_filter = FlowRunFilter(
start_time=FlowRunFilterStartTime(before_=cutoff),
state=FlowRunFilterState(
type=FlowRunFilterStateType(any_=[StateType.COMPLETED, StateType.FAILED, StateType.CANCELLED])
),
)
total = 0
while True:
runs = await client.read_flow_runs(flow_run_filter=fr_filter, limit=batch_size)
if not runs:
break
for fr in runs:
try:
await client.delete_flow_run(fr.id)
total += 1
except Exception as e:
print(f"Failed to delete {fr.id}: {e}")Marvin
11/27/2025, 11:24 AM- Delete a single flow run from CLI (verified):
prefect flow-run delete --help
prefect flow-run delete <flow-run-id>
- Reclaim disk space in SQLite after deletions:
sqlite3 ~/.prefect/prefect.db "VACUUM; ANALYZE;"
Verify current settings
prefect config view --show-defaults --hide-sources | grep -E "EVENTS_|LOGGING_TO_API|LOGGING_LEVEL|EVENT_PERSISTER"
```
What I suspect in your case
- With only ~260 runs, 96 MB is usually events + logs. If your flows log at INFO and logging-to-API is enabled, the log table alone can grow quickly. And if events retention is left at 7 days, the events table tends to be the other top contributor.
A couple of quick wins
- Set PREFECT_LOGGING_TO_API_ENABLED=false (or raise PREFECT_LOGGING_LEVEL to WARNING) to cut log growth
- Set PREFECT_API_EVENTS_RETENTION_PERIOD="2d" to keep the events table small
- Run the cleanup script to purge old runs, then VACUUM
Questions to tailor this further
- Are you running the default local SQLite or Postgres?
- Do you need logs visible in the UI, or can we disable API logging?
- How long do you need event history retained?
If you’d like, paste the results of the SQL row counts above and I’ll point out the exact “top talker” tables and the minimal changes to keep your DB small.Nate
11/27/2025, 4:04 PM