<@ULVA73B9P> how to clear older runs and log from ...
# marvin-ai
j
@Marvin how to clear older runs and log from postgres db
m
thought for 208.6 seconds
Thanks for asking! A few quick clarifiers to make sure I point you the right way: - Are you running self-hosted Prefect Server (OSS) with Postgres? - Do you want to delete just logs, or both flow/task runs and their logs? - What retention window do you want (e.g., “older than 30 days”)? Assuming Prefect 3.x self-hosted with Postgres and you want to purge “older than N days,” here are the supported approaches. Option A (recommended): Use the Prefect API/Client to delete runs (logs go with them) - Deleting a flow run via the API removes the run and its related data (task runs, states, and associated logs). - This is the safest, schema-aware approach. Example: bulk-delete flow runs older than 30 days with the Python client - This uses filters to find runs before a cutoff and deletes them. You can schedule this as a maintenance flow to run nightly/weekly.
Copy code
import asyncio
from datetime import datetime, timedelta, timezone

from prefect.client.orchestration import get_client
from prefect.client.schemas import filters, sorting

# Configure how far back to retain
DAYS_TO_KEEP = 30

async def main():
    cutoff = datetime.now(timezone.utc) - timedelta(days=DAYS_TO_KEEP)
    async with get_client() as client:
        # Page through runs to avoid loading everything at once
        page_size = 500
        page = 1
        total_deleted = 0

        while True:
            runs = await client.read_flow_runs(
                flow_runs_filter=filters.FlowRunFilter(
                    # only runs that started before cutoff (you could also use end_time)
                    start_time=filters.FlowRunFilterStartTime(before=cutoff),
                    # optional: skip active runs
                    state_type=filters.FlowRunFilterStateType(any_=["COMPLETED", "FAILED", "CANCELLED", "CRASHED"])
                ),
                sort=sorting.FlowRunSort.START_TIME_ASC,
                limit=page_size,
                offset=(page-1)*page_size,
            )

            if not runs:
                break

            for fr in runs:
                try:
                    await client.delete_flow_run(fr.id)
                    total_deleted += 1
                except Exception as exc:
                    print(f"Failed to delete {fr.id}: {exc}")

            page += 1

        print(f"Deleted {total_deleted} flow runs older than {DAYS_TO_KEEP} days.")

if __name__ == "__main__":
    asyncio.run(main())
- Make sure your script can reach your server (e.g.,
PREFECT_API_URL
is set). - Docs: API reference: flow_runs and Filters One-off via CLI (per-ID): - We verified this command exists:
Copy code
prefect flow-run delete <FLOW_RUN_ID>
- You can list runs to find IDs:
Copy code
prefect flow-run ls --limit 100
- There isn’t a built-in bulk delete in the CLI, so for large purges use the Python client script above. Option B: Delete only logs (keep runs) - There isn’t a public CLI command to delete logs only. While the server’s internal models support log deletion, there’s no public endpoint exposed for it in the API module. - If you truly need to keep runs but drop older logs, you can do it directly in Postgres. Proceed with caution and take a backup first. - The exact table name may be
log
or
logs
depending on version; you can check which exists:
Copy code
SELECT to_regclass('public.log') AS log_exists, to_regclass('public.logs') AS logs_exists;
- Example (adjust table name and timestamp column as found in your schema; typically
log
and
timestamp
):
Copy code
BEGIN;
-- Delete logs older than 30 days
DELETE FROM log WHERE timestamp < NOW() - INTERVAL '30 days';
-- Reclaim space and update stats (optional but recommended)
VACUUM ANALYZE log;
COMMIT;
- If you get FK violations, delete in this order: logs first, then run states, then task runs, then flow runs — or stick to Option A which handles relationships for you. - Internal reference for log capabilities: Server models: logs Option C: Nuke everything (dev-only) - If you just want a clean slate in a dev environment:
Copy code
prefect server database reset
- Verified command group:
Copy code
prefect server database --help
Notes and best practices - There is no out-of-the-box “retention policy” toggle in Prefect 3.x for auto-purging. Most teams schedule a maintenance flow using Option A. - Test on a staging DB or with a small cutoff first. - Back up your database before large deletes. If you share: - Your Prefect version (2.x vs 3.x), - Whether you want to remove runs+logs or logs only, - Your retention window, I can tailor a ready-to-run script or SQL for your setup.