Great question — keeping your Prefect 3 Postgres DB in check comes down to three levers: - Enable Prefect’s built-in vacuum/retention service - Set sensible event retention - Do periodic Postgres maintenance (VACUUM/monitoring), with an optional custom cleanup flow for fine-grained control Below is a concise setup you can apply. I’m assuming Prefect 3.x and a self-hosted Postgres. 1) Turn on Prefect’s DB vacuum service (housekeeping) The vacuum service deletes old flow runs and cleans up orphaned logs/artifacts. It also prunes old events. - Enable services and set retention

# Enable both event and flow run cleanup
export PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED="events,flow_runs"

# How often the cleanup runs (seconds)
export PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS=3600

# Delete terminal flow runs older than 30 days
export PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD=2592000

# Batch size per transaction
export PREFECT_SERVER_SERVICES_DB_VACUUM_BATCH_SIZE=200

You can set these via profiles too:

prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED="events,flow_runs"
prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS=3600
prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD=2592000
prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_BATCH_SIZE=200

- Ensure background services are running

# If you run everything in one process
prefect server start

# If you run API and services separately, make sure services are up
prefect server services start
prefect server services ls

Docs: Database maintenance & vacuum service 2) Shorten event retention (big win for DB size) Events can be your largest tables. Set a shorter retention for high-volume workloads.

# Keep 2 days of events
export PREFECT_EVENTS_RETENTION_PERIOD="2d"
# or
prefect config set PREFECT_EVENTS_RETENTION_PERIOD="2d"

Docs: Settings reference (see

PREFECT_EVENTS_RETENTION_PERIOD

) 3) Optional: custom cleanup flow (fine-grained control) If you want to explicitly delete old terminal runs (e.g., by tag/app), schedule a small retention flow as a deployment:

import anyio
from datetime import datetime, timedelta, timezone
from prefect import flow, get_run_logger
from prefect.client.orchestration import get_client
from prefect.client.schemas.filters import (
    FlowRunFilter,
    FlowRunFilterState,
    FlowRunFilterStateType,
    FlowRunFilterStartTime,
)
from prefect.client.schemas.objects import StateType

@flow
async def prune_old_runs(days: int = 30, limit: int = 500):
    logger = get_run_logger()
    cutoff = datetime.now(timezone.utc) - timedelta(days=days)
    fr_filter = FlowRunFilter(
        start_time=FlowRunFilterStartTime(before_=cutoff),
        state=FlowRunFilterState(
            type=FlowRunFilterStateType(
                any_=[StateType.COMPLETED, StateType.FAILED, StateType.CANCELLED, StateType.CRASHED]
            )
        ),
    )
    async with get_client() as client:
        runs = await client.read_flow_runs(flow_run_filter=fr_filter, limit=limit)
        <http://logger.info|logger.info>(f"Deleting {len(runs)} flow runs older than {days} days")
        for r in runs:
            await client.delete_flow_run(r.id)

if __name__ == "__main__":
    anyio.run(prune_old_runs)

API reference: Prefect orchestration client 4) Postgres maintenance and monitoring Even with deletions, Postgres needs to reclaim space. - Check sizes and bloat ``` -- DB size SELECT pg_size_pretty(pg_database_size('prefect')) AS database_size; -- Largest tables SELECT relname AS table, pg_size_pretty(pg_total_relation_size(relid)) AS total_size, n_live_tup AS rows FROM pg_catalog.pg_statio_user_tables JOIN pg_stat_user_tables USING (relid) ORDER BY pg_total_relation_size(relid) DESC LIMIT 20;

-- Dead tuples (bloat signal) SELECT relname AS table, n_live_tup, n_dead_tup, last_autovacuum FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 20;

- Run safe vacuum/analyze (non-blocking)

VACUUM ANALYZE flow_run; VACUUM ANALYZE task_run; VACUUM ANALYZE log; VACUUM ANALYZE events;

- Consider per-table autovacuum tuning for high-volume tables (e.g., events)

ALTER TABLE events SET ( autovacuum_vacuum_scale_factor = 0.05, autovacuum_vacuum_threshold = 1000, autovacuum_analyze_scale_factor = 0.02 ); ``` Full guidance: Database maintenance guide Quick checklist - Enable vacuum service for

events,flow_runs

- Set

PREFECT_EVENTS_RETENTION_PERIOD

to 1–3 days if you have high volume - Confirm services are running:

prefect server services ls

- Monitor table sizes and bloat; run VACUUM ANALYZE as needed - Optionally, schedule a custom “prune” flow for targeted cleanup If you can share your workload profile (runs/day, DB size now, deployment model: single process vs. split services, Kubernetes/Docker), I can suggest concrete retention values and autovacuum settings tailored to your setup.

Got it — if the log table is the main space hog, here are your options to keep it under control in Prefect 3: What Prefect cleans up automatically - There is no separate “log retention” setting. - Logs are deleted when their flow run is deleted. - The db vacuum service will clean orphaned logs and delete logs indirectly when it deletes old flow runs — it will not prune logs by age on their own. Recommended approaches A) Use flow-run retention to prune logs automatically - Turn on the flow-run vacuum and set a shorter retention. This deletes terminal flow runs and their logs.

prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED="events,flow_runs"
prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD=2592000  # 30 days
prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS=3600
prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_BATCH_SIZE=200

# Ensure services are running
prefect server services start
prefect server services ls

Docs: Database maintenance & vacuum service B) If you want to keep runs but prune logs faster, schedule a SQL cleanup - Since logs don’t have an age-based retention, run a small scheduled job (cron, pg_cron, or a Prefect flow) that deletes old rows by timestamp and then VACUUMs the table. - Example SQL you can run on a schedule:

-- Delete logs older than 14 days in chunks to avoid long locks
-- Repeat until 0 rows deleted
WITH to_delete AS (
  SELECT ctid FROM log
  WHERE timestamp < NOW() - INTERVAL '14 days'
  LIMIT 10000
)
DELETE FROM log
WHERE ctid IN (SELECT ctid FROM to_delete);

-- Reclaim space and update statistics
VACUUM ANALYZE log;

- If you prefer to orchestrate this with Prefect, create a tiny “maintenance” flow that runs this SQL (via psycopg/asyncpg) and deploy it on a schedule. C) Slow future growth (if acceptable) - Lower log volume by reducing the client logging level or skipping API logging entirely:

# Reduces what gets emitted
prefect config set PREFECT_LOGGING_LEVEL="WARNING"

# If you don’t need logs in the DB at all
prefect config set PREFECT_LOGGING_TO_API_ENABLED=false

Note: turning off API logging means those logs won’t be visible in the UI. Postgres tips for the log table - After large deletes, run:

VACUUM ANALYZE log;

- Consider more aggressive autovacuum just for logs:

ALTER TABLE log SET (
  autovacuum_vacuum_scale_factor = 0.05,
  autovacuum_vacuum_threshold = 5000,
  autovacuum_analyze_scale_factor = 0.02
);

- Monitor growth:

SELECT pg_size_pretty(pg_total_relation_size('public.log')) AS log_size, COUNT(*) AS rows FROM log;

SELECT DATE(timestamp) AS day, COUNT(*) AS logs_per_day
FROM log
GROUP BY DATE(timestamp)
ORDER BY day DESC
LIMIT 30;

Quick decision guide - If you’re okay deleting old runs: enable flow-run vacuum with a shorter retention — logs will go with them automatically. - If you must keep runs but trim logs: schedule a SQL cleanup job to delete old logs by timestamp, then VACUUM. If you share your desired log retention window (e.g., 7–14 days) and whether you can shorten flow-run retention, I can provide a drop-in Prefect flow that batches deletions safely for your setup.

Short answer: there isn’t a Prefect config setting for “log retention.” In Prefect 3, logs don’t have their own retention policy — they’re deleted when their parent flow runs are deleted, and the db vacuum service only removes orphaned logs and logs that belong to flow runs it deletes. How to enforce 30-day log retention Option 1 — If you can delete flow runs after 30 days - Enable the flow-run vacuum and set its retention to 30 days. This will delete terminal flow runs older than 30 days and their logs with them.

prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED="events,flow_runs"
prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD=2592000  # 30 days in seconds
prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS=3600
prefect config set PREFECT_SERVER_SERVICES_DB_VACUUM_BATCH_SIZE=200

prefect server services start
prefect server services ls

Docs: Database maintenance & vacuum service Option 2 — Keep runs longer, but prune logs after 30 days - Schedule a small SQL job (cron, pg_cron, or a Prefect flow) that deletes log rows older than 30 days in batches, then VACUUMs the table. Example SQL:

-- Index to speed filtering by timestamp (safe to run once)
CREATE INDEX IF NOT EXISTS ix_log_timestamp ON log (timestamp);

-- Delete logs older than 30 days in chunks
WITH to_delete AS (
  SELECT ctid
  FROM log
  WHERE timestamp < NOW() - INTERVAL '30 days'
  LIMIT 10000
)
DELETE FROM log
WHERE ctid IN (SELECT ctid FROM to_delete);

-- Repeat the DELETE until it affects 0 rows

-- Reclaim space and update stats (run outside a transaction)
VACUUM ANALYZE log;

If you want to orchestrate this with Prefect, here’s a drop-in flow you can schedule. It deletes in batches and then runs VACUUM in autocommit mode:

import os
import time
from prefect import flow, task, get_run_logger
import psycopg

@task
def delete_old_logs(dsn: str, days: int = 30, batch: int = 10000) -> int:
    total = 0
    with psycopg.connect(dsn) as conn:
        # Optional: reduce risk of long-running statements
        with conn.cursor() as cur:
            cur.execute("SET statement_timeout TO '5min'")
        while True:
            with conn.cursor() as cur:
                cur.execute("""
                    WITH to_delete AS (
                        SELECT ctid
                        FROM log
                        WHERE timestamp < NOW() - INTERVAL %s
                        LIMIT %s
                    )
                    DELETE FROM log
                    WHERE ctid IN (SELECT ctid FROM to_delete)
                """, (f"{days} days", batch))
                deleted = cur.rowcount or 0
            conn.commit()
            total += deleted
            if deleted == 0:
                break
            time.sleep(0.1)
    return total

@task
def vacuum_log_table(dsn: str):
    # VACUUM must run in autocommit
    with psycopg.connect(dsn, autocommit=True) as conn:
        with conn.cursor() as cur:
            cur.execute("VACUUM ANALYZE log;")

@flow
def prune_prefect_logs(dsn: str = os.getenv("PREFECT_DB_DSN", ""), days: int = 30, batch: int = 10000):
    logger = get_run_logger()
    deleted = delete_old_logs(dsn, days, batch)
    vacuum_log_table(dsn)
    <http://logger.info|logger.info>(f"Deleted {deleted} log rows older than {days} days")

if __name__ == "__main__":
    prune_prefect_logs()

Notes and tips - There’s no REST API or config for “delete logs older than N days”; it’s done indirectly via flow-run deletion or directly via SQL as shown. - Keep event retention short too (events are separate from logs): -

prefect config set PREFECT_EVENTS_RETENTION_PERIOD="7d"

- After large deletions, always run

VACUUM ANALYZE log;

(autocommit) or ensure autovacuum is tuned: -

ALTER TABLE log SET (autovacuum_vacuum_scale_factor = 0.05, autovacuum_vacuum_threshold = 5000, autovacuum_analyze_scale_factor = 0.02);

Would you like me to tailor a ready-to-run maintenance deployment for your environment? If you can share whether you can delete flow runs after 30 days (or must retain them longer), I’ll recommend the best path and exact settings.

You’re seeing

/tmp/runner_storage

because the Prefect Runner (used by

prefect worker start

) creates a per-runner temp directory under the system temp dir. By default that’s whatever

tempfile.gettempdir()

returns (usually

/tmp

on Linux). There isn’t a Prefect setting to point this elsewhere, but you can change it by setting the OS temp directory env var before starting the worker: - Linux/macOS (POSIX):

export TMPDIR=/mnt/prefect-tmp
mkdir -p "$TMPDIR"
prefect worker start --pool <your-pool>

- Windows (PowerShell):

$env:TEMP="D:\prefect-tmp"
$env:TMP="D:\prefect-tmp"
prefect worker start --pool <your-pool>

- systemd unit example:

[Service]
Environment=TMPDIR=/mnt/prefect-tmp
ExecStart=/usr/local/bin/prefect worker start --pool <your-pool>

- Docker (running the worker in a container):

docker run --rm \
  -e TMPDIR=/mnt/prefect-tmp \
  -v /bigger-disk/prefect-tmp:/mnt/prefect-tmp \
  prefecthq/prefect:3-latest \
  prefect worker start --pool <your-pool>

- Kubernetes (for a process worker pod): - Add an env var

TMPDIR=/mnt/prefect-tmp

to the worker container - Mount an

emptyDir

or PVC at

/mnt/prefect-tmp

Notes - Each runner instance creates a unique subfolder:

<TMPDIR>/runner_storage/<uuid>

, and Prefect cleans it up when the worker exits. - If you’re actually using Docker or Kubernetes work pools to run flows in containers/pods, also set

TMPDIR

in the job template for the flow-run container so the engine inside that container uses your desired location too (and mount storage there). If you’d like, share how you’re running the worker (systemd, Docker, K8s) and I can provide a ready-to-copy snippet for that environment.