<@ULVA73B9P> After only a few days my prefect.db h...
# ask-marvin
c
@Marvin After only a few days my prefect.db has become 96 megabytes. Only 260 flow runs have been made. I need to figure out what is causing all of this storage
m
thought for 425.3 seconds
Got it — let’s quickly pinpoint what’s filling your local SQLite
prefect.db
and trim it. Assuming Prefect 3.x with the default local server and SQLite: - The biggest space users are usually: - events (high-volume; default retention 7 days) - logs (all flow/task logs sent to the API; no default retention) - run state history (flow_run_state/task_run_state) First, let’s confirm what’s actually large in your DB Run these against your SQLite DB (typically at
~/.prefect/prefect.db
):
Copy code
-- Row counts for key tables
SELECT 'events' AS t, COUNT(*) FROM events
UNION ALL SELECT 'log', COUNT(*) FROM log
UNION ALL SELECT 'flow_run_state', COUNT(*) FROM flow_run_state
UNION ALL SELECT 'task_run_state', COUNT(*) FROM task_run_state
UNION ALL SELECT 'flow_run', COUNT(*) FROM flow_run
UNION ALL SELECT 'artifact', COUNT(*) FROM artifact
ORDER BY t;

-- Approx content size by summing text/json column lengths
SELECT 'log_bytes' AS what, IFNULL(SUM(LENGTH(message)),0) FROM log
UNION ALL SELECT 'events_bytes', IFNULL(SUM(
  LENGTH(event) + COALESCE(LENGTH(resource),0) + COALESCE(LENGTH(related),0) + COALESCE(LENGTH(payload),0)
),0) FROM events;

-- How far back your data goes (helps confirm retention)
SELECT 'events' AS t, MIN(occurred), MAX(occurred) FROM events
UNION ALL SELECT 'log', MIN(timestamp), MAX(timestamp) FROM log
UNION ALL SELECT 'flow_run', MIN(created), MAX(created) FROM flow_run;

-- DB stats
PRAGMA page_count;
PRAGMA page_size;
PRAGMA freelist_count;
PRAGMA integrity_check;
Common findings you may see: - Lots of events for relatively few runs (each state change emits events) - Many logs if your flows default to INFO logging and send logs to API - SQLite file size won’t shrink after deletes until you run VACUUM Quick ways to reduce growth going forward - Lower event retention (server setting; default ~7 days) - Use a tighter window like 2–3 days if you don’t need long-lived event history:
Copy code
prefect config set PREFECT_API_EVENTS_RETENTION_PERIOD="2d"
Restart the API; events older than the retention will be batch-trimmed automatically. - Reduce or disable logging to the API (client setting) - If you do not need logs in the UI/API:
Copy code
prefect config set PREFECT_LOGGING_TO_API_ENABLED=false
- Or at least reduce verbosity:
Copy code
prefect config set PREFECT_LOGGING_LEVEL="WARNING"
This dramatically cuts DB writes to the
log
table. - Optional server-side tuning for event cleanup throughput (helps large backlogs clear faster):
Copy code
prefect config set PREFECT_API_SERVICES_EVENT_PERSISTER_BATCH_SIZE="50"
  prefect config set PREFECT_API_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL="3"
Then restart the API. Clean up existing data now - Delete old flow runs (and their states, artifacts, logs) programmatically: ``` import asyncio from datetime import datetime, timedelta, timezone from prefect.client.orchestration import get_client from prefect.client.schemas.filters import FlowRunFilter, FlowRunFilterStartTime, FlowRunFilterState, FlowRunFilterStateType from prefect.states import StateType async def delete_old_flow_runs(days_to_keep=14, batch_size=100): cutoff = datetime.now(timezone.utc) - timedelta(days=days_to_keep) async with get_client() as client: fr_filter = FlowRunFilter( start_time=FlowRunFilterStartTime(before_=cutoff), state=FlowRunFilterState( type=FlowRunFilterStateType(any_=[StateType.COMPLETED, StateType.FAILED, StateType.CANCELLED]) ), ) total = 0 while True: runs = await client.read_flow_runs(flow_run_filter=fr_filter, limit=batch_size) if not runs: break for fr in runs: try: await client.delete_flow_run(fr.id) total += 1 except Exception as e: print(f"Failed to delete {fr.id}: {e}")
🙌 1
print(f"Deleted {total} flow runs") asyncio.run(delete_old_flow_runs())
Copy code
- Delete a single flow run from CLI (verified):
prefect flow-run delete --help prefect flow-run delete <flow-run-id>
Copy code
- Reclaim disk space in SQLite after deletions:
sqlite3 ~/.prefect/prefect.db "VACUUM; ANALYZE;"
Copy code
Verify current settings
prefect config view --show-defaults --hide-sources | grep -E "EVENTS_|LOGGING_TO_API|LOGGING_LEVEL|EVENT_PERSISTER" ``` What I suspect in your case - With only ~260 runs, 96 MB is usually events + logs. If your flows log at INFO and logging-to-API is enabled, the
log
table alone can grow quickly. And if events retention is left at 7 days, the
events
table tends to be the other top contributor. A couple of quick wins - Set
PREFECT_LOGGING_TO_API_ENABLED=false
(or raise
PREFECT_LOGGING_LEVEL
to WARNING) to cut log growth - Set
PREFECT_API_EVENTS_RETENTION_PERIOD="2d"
to keep the events table small - Run the cleanup script to purge old runs, then
VACUUM
Questions to tailor this further - Are you running the default local SQLite or Postgres? - Do you need logs visible in the UI, or can we disable API logging? - How long do you need event history retained? If you’d like, paste the results of the SQL row counts above and I’ll point out the exact “top talker” tables and the minimal changes to keep your DB small.