As for managing concurrency, have you already explored setting deployment-level concurrency?

Hey @Marvin in prefect 2.20, what's the default timeout for a

run_deployment

call? if that timeout is reached while running, is it really crashing or does the job keep on running and once succeeds it's all good?

Yep—I'm here and ready to help! What are you working on with Prefect today? If you’ve got an error, code snippet, or goal you’re aiming for, drop it in and I’ll jump in.

Awesome—glad it worked out! If you want to add notifications or a quick check to catch this sooner next time, I’m happy to help. Otherwise, you’re all set.

Short answer: there isn’t a safe way to “poke” a running parent flow in Prefect 2.x to rebind to subflows you re-ran manually. The parent is only waiting on the specific child flow run IDs it launched. If you retriggered children as new runs, the parent won’t ever see those complete. Immediate actions to stop spend - Cancel the parent flow run now (this will tear down the pod):

prefect flow-run cancel <PARENT_FLOW_RUN_ID>

You can also click “Cancel” in the UI on the parent run. If the pod doesn’t terminate promptly after cancelling, manually delete the Kubernetes job/pod. Why this happened - If your parent launched child runs via

run_deployment(...)

or

create_flow_run

and then waited on them, it’s waiting on those specific run IDs. - Manually re-running a child from the UI often creates a new run; the parent won’t notice that new run finishing. - One more gotcha:

wait_for_flow_run

has a default 3h timeout. If your parent used it without overriding

timeout

, it may have stopped waiting after 3 hours. What to do next - If you only need the parent marked done (and don’t need wrap-up code to execute), cancelling is the cleanest way to stop the pod. Starting a fresh “wrap-up” flow that checks downstream results (and does any finalization) is safer than trying to force the parent to continue. - If you do need the parent’s wrap-up code to run, re-run the parent with logic that skips work for children that are already Completed. Patterns to use going forward 1) Call subflows as functions (best for tight orchestration)

from prefect import flow

@flow
def child(item_id: str):
    ...

@flow
def parent(ids: list[str]):
    for i in ids:
        child.with_options(name=f"child-{i}")(i)  # waits for each child

2) If you use deployments, explicitly wait for the exact child runs you started and remove the timeout

from prefect import flow
from prefect.deployments import run_deployment
from prefect.flow_runs import wait_for_flow_run

@flow
def parent(ids: list[str]):
    runs = []
    for i in ids:
        fr = run_deployment(
            name="child-flow/deployment",
            parameters={"id": i},
            as_subflow=True,   # ties the run to this parent
            timeout=None       # do not stop waiting early
        )
        runs.append(fr)

    for fr in runs:
        wait_for_flow_run(fr.id, timeout=None, log_states=True)

3) Prefer configuring retries on the child flow (so the same run ID retries) instead of manually re-running

@flow(retries=3, retry_delay_seconds=60)
def child(...):
    ...

A couple questions so I can give you targeted fixes: - How did the parent launch the subflows: direct

@flow

calls or

run_deployment

? - How did you retrigger the children: UI “Retry” on the failed run, or “Run”/“Rerun” on the deployment (creating new runs)? - Are you using a Kubernetes worker/pool for the parent pod? If you share those details or a minimal snippet of the parent orchestration, I can suggest the smallest change to make this recoverable and prevent it in the future.

Short answer: there isn’t a global “disable all schedules” toggle in the Prefect 2.20 UI. What you can do instead (choose based on your goal): If you just want to stop compute (but don’t mind runs still being scheduled) - Pause your work pools (fewer things to click than hundreds of deployments) - UI: Pause each work pool in the Work Pools page - CLI:

prefect work-pool pause "<pool-name>"

- You can also stop/scale down your workers. This prevents any new runs from executing. - Docs: Work pools If you want to stop new scheduled runs from being created - Self-hosted Prefect Server only: disable the scheduler service globally and restart the API - Set env var on your Prefect API service:

PREFECT_API_SERVICES_SCHEDULER_ENABLED=false

- This stops all scheduled flow runs from being created until you re-enable it. - Prefect Cloud: there’s no workspace-level “disable scheduler” switch. You’ll need to pause/clear schedules per deployment: - List schedules for a deployment:

prefect deployment schedule ls "<deployment-name>"

- Pause a specific schedule:

prefect deployment schedule pause "<deployment-name>" <schedule-id>

- Remove all schedules for a deployment (you can re-create them by re-deploying later):

prefect deployment schedule clear "<deployment-name>"

- Schedules docs: Schedules in Prefect 2 Notes on bulk operations - There isn’t a one-click bulk pause/clear for all deployments in 2.20. - Practical approaches: - If you have a small number of work pools, pausing pools is the fastest way to stop spend immediately. - If you need to truly stop new runs from being created, you can script across deployments with the CLI (e.g., iterate over deployment names and call

prefect deployment schedule clear "<deployment>"

). I’m happy to provide a small script if you can share how your deployments are named or whether you’re on Mac/Linux/Windows. Quick clarifications to tailor the steps: 1) Are you on Prefect Cloud or self-hosted Prefect Server? 2) Do you want to stop runs from being created at all, or just prevent execution to save compute?

You’re reading the UI right: - The top toggle is the deployment-level state (Active vs Paused). If a deployment is paused, no schedules for it will create runs. - The toggles under the Schedules section are per-schedule. Each schedule must be enabled to emit runs. - For scheduled runs to be created, the deployment must be Active AND at least one schedule under it must be enabled. There isn’t a single UI filter for “both on” in 2.20, but we can query this programmatically. Two options: Option A: Python (uses the Prefect 2 client to find deployments where paused=False and at least one schedule is active) - Works on both Prefect Cloud and self-hosted, respects your PREFECT_API_URL/PREFECT_API_KEY. If you want, I can drop in a ready-to-run snippet; just let me know: - Are you on Prefect Cloud or self-hosted? - Do you prefer a Python script or a quick-and-dirty shell loop? Option B: CLI + loop (quick check without coding) - Iterate all deployments, skip those paused, and check if any schedule is active. - This is a bit brittle due to table parsing, but I can provide a bash one-liner if that’s easier for your environment. References - Schedules in Prefect 2 - CLI schedule management: - List schedules for a deployment:

prefect deployment schedule ls "<deployment-name>"

- Pause a specific schedule:

prefect deployment schedule pause "<deployment-name>" <schedule-id>

- Clear all schedules for a deployment:

prefect deployment schedule clear "<deployment-name>"

Great — here’s a ready-to-run Python script that lists all deployments where: - the deployment itself is Active (not paused), and - it has at least one enabled schedule. It works with Prefect 2.20 (Cloud or self-hosted). It uses your configured PREFECT_API_URL and PREFECT_API_KEY.

import asyncio
from typing import List, Dict

try:
    # Prefect 2.10+ exposes get_client here
    from prefect.client import get_client
except ImportError:
    # Older 2.x fallback
    from prefect.client.orion import get_client  # type: ignore


async def find_active_deployments_with_enabled_schedules() -> List[Dict]:
    rows = []
    async with get_client() as client:
        offset = 0
        limit = 200

        while True:
            deployments = await client.read_deployments(offset=offset, limit=limit)
            if not deployments:
                break

            for d in deployments:
                # Deployment must be active
                if getattr(d, "paused", False):
                    continue

                # Get full deployment so we can inspect schedules
                full = await client.read_deployment(d.id)

                schedules = getattr(full, "schedules", []) or []
                active_schedules = [s for s in schedules if getattr(s, "active", False)]
                if not active_schedules:
                    continue

                flow_name = getattr(full, "flow_name", None)
                name = f"{flow_name}/{full.name}" if flow_name else full.name
                rows.append(
                    {
                        "name": name,
                        "deployment_id": str(full.id),
                        "work_pool": getattr(full, "work_pool_name", None)
                                      or getattr(full, "work_queue_name", None),
                        "active_schedule_count": len(active_schedules),
                        # Optional: include schedule IDs/times
                        "schedule_ids": [str(s.id) for s in active_schedules],
                    }
                )

            offset += limit

    return rows


if __name__ == "__main__":
    results = asyncio.run(find_active_deployments_with_enabled_schedules())
    print(f"Deployments active with enabled schedules: {len(results)}")
    for r in results:
        print(
            f"- {r['name']} "
            f"(id={r['deployment_id']}, pool={r['work_pool']}, "
            f"active_schedules={r['active_schedule_count']}, "
            f"schedule_ids={r['schedule_ids']})"
        )

If you’d like, I can tweak this to: - dump to CSV/JSON, - include schedule expressions/next times, - or directly pause/clear schedules for everything it finds. Reference: Schedules in Prefect 2

Short answer: your server’s scheduler is falling behind because its queries to Postgres are timing out. That warning plus the asyncpg TimeoutError means the RecentDeploymentsScheduler is spending 20s+ inside DB calls each cycle (its loop defaults to 5s), which matches your RDS cost spike — the DB is under sustained load doing scheduling work. Likely culprits - High‑frequency schedules (especially RRule patterns like MINUTELY or BYSECOND) that expand a lot of run times each loop; these are known to be costly - See: Scheduler CPU spikes and “RecentDeploymentsScheduler took … longer than loop interval” with RRule schedules - A large scheduling horizon and/or many deployments, causing large batches of run generations per loop - Database contention or insufficient resources (small instance, connection pool mismatch, table bloat/vacuuming) - Multiple Prefect API/services processes each maintaining their own DB pools, multiplying your total connections Quick mitigations to try - If you have any second/minute‑level RRule schedules, switch to Cron/Interval where possible or reduce frequency. These are the most common source of heavy scheduling work. - Reduce the scheduler’s per‑loop work and scheduling horizon: - Increase the loop interval for “recent deployments” - Lower batch sizes and max runs - Narrow the scheduling window (bigger min_scheduled_time and/or smaller max_scheduled_time) Example env vars you can set for the server services container

# Slow the loops, reduce work per cycle
PREFECT_SERVER_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS=30
PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS=60
PREFECT_SERVER_SERVICES_SCHEDULER_MAX_RUNS=50
PREFECT_SERVER_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE=50
PREFECT_SERVER_SERVICES_SCHEDULER_INSERT_BATCH_SIZE=250

# Narrow the scheduling horizon (defaults: min=PT1H, max=P100D)
PREFECT_SERVER_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME=PT2H
PREFECT_SERVER_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME=P7D

DB tuning knobs (per server/services process) - Pool sizing: larger pools reduce connection wait, but raise total RDS connections (and cost). Tune carefully.

PREFECT_SERVER_DATABASE_POOL_SIZE=10
PREFECT_SERVER_DATABASE_MAX_OVERFLOW=10
PREFECT_SERVER_DATABASE_POOL_TIMEOUT=30
PREFECT_SERVER_DATABASE_POOL_RECYCLE=3600
PREFECT_SERVER_DATABASE_CONNECT_ARGS__APPLICATION_NAME=prefect-server
# Optional: reduce prepared/statement cache sizes if memory pressure is high
PREFECT_SERVER_DATABASE_CONNECT_ARGS__PREPARED_STATEMENT_CACHE_SIZE=100
PREFECT_SERVER_DATABASE_CONNECT_ARGS__STATEMENT_CACHE_SIZE=100

Settings reference: Prefect 3 settings reference Database side checks - Enable slow query logging (e.g., log_min_duration_statement) or use RDS Performance Insights to confirm which statements are hot; look for scheduling queries hitting deployments and schedules tables. - Ensure autovacuum is effective; vacuum/analyze if there’s bloat. - If needed, increase DB instance size/IO temporarily to clear backlog. - Consider setting a Postgres statement_timeout at the DB/parameter-group level to fail fast on pathological queries. Upgrade note - If you’re not on the latest 3.x, upgrading is recommended — we’ve shipped numerous scheduler and DB performance improvements over time. Release notes: Prefect releases A few clarifying questions to pinpoint this faster - Which Prefect version are you on? - Roughly how many deployments and which schedule types are you using? Any RRule schedules with MINUTELY/BYSECOND? - How many Prefect API/services processes are running (each one has its own connection pool)? - Any recent growth in flow_runs or events volume around when the RDS cost spiked? If you can share your scheduler-related env settings and a sample of the schedule definitions that run most frequently, I can suggest a tighter set of values tailored to your load.

Thanks for confirming — got it, you’re on Prefect 2.20. Why you didn’t see those env vars - That makes sense: in 2.x most scheduler/database settings are optional. If you don’t set the environment variables, Prefect uses built‑in defaults and nothing will show up in the pod’s env. - Also, 2.x and 3.x use different env var prefixes. In 2.x they start with `PREFECT_API_...`; in 3.x they start with

PREFECT_SERVER_...

. If you were looking for

PREFECT_SERVER_*

keys in a 2.x server, they won’t exist. Yes, the log name is expected - The log component

prefect.server.services.recentdeploymentsscheduler

exists in 2.x as well. The warning “RecentDeploymentsScheduler took X seconds…” means that pass of scheduling took longer than its loop interval (the “recent deployments” loop defaults to 5 seconds in 2.x). Prefect 2.20 scheduler defaults and env var names - These are the common 2.x knobs and their defaults. If you don’t set the env vars, Prefect uses these defaults: Scheduler service (2.x) -

PREFECT_API_SERVICES_SCHEDULER_ENABLED

= true -

PREFECT_API_SERVICES_SCHEDULER_LOOP_SECONDS

= 60 -

PREFECT_API_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS

= 5 -

PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE

= 100 -

PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE

= 500 -

PREFECT_API_SERVICES_SCHEDULER_MAX_RUNS

= 100 -

PREFECT_API_SERVICES_SCHEDULER_MIN_RUNS

= 3 -

PREFECT_API_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME

= PT1H (1 hour) -

PREFECT_API_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME

= P100D (100 days) Database pool (2.x) -

PREFECT_API_DATABASE_POOL_SIZE

= 5 -

PREFECT_API_DATABASE_MAX_OVERFLOW

= 10 -

PREFECT_API_DATABASE_POOL_TIMEOUT

= 30 -

PREFECT_API_DATABASE_POOL_RECYCLE

= 3600 -

PREFECT_API_DATABASE_CONNECTION_URL

= your DSN (this is the one you typically have set) How to view the effective config (with defaults) - Exec into the API/server pod and run:

prefect config view --show-defaults | grep -i -e api.services.scheduler -e api.database -e SCHEDULER -e DATABASE

- You can also narrow in on scheduler:

prefect config view --show-defaults | sed -n '/api.services.scheduler/,+25p'

Why the warning + RDS cost spike - In 2.20 this usually means the scheduler’s DB queries are slow/expensive — often due to frequent schedules (e.g., RRule MINUTELY/BYSECOND), a wide scheduling horizon, many deployments, or DB resource/connection pressure. Each pass tries to generate upcoming runs; if that work takes > loop interval, it falls behind and keeps hammering the DB. If you want, share: - How many deployments and which schedule types you have (especially any RRule schedules) - Your DB instance size and observed connections - Whether you’re running multiple API/services pods (each has its own pool) I can suggest concrete 2.x tuning values to reduce load. Also, if/when you’re ready, upgrading to 3.x includes additional scheduler and DB performance improvements; I’m happy to outline a path when it’s helpful. Refs - 2.x settings guide: Prefect 2 settings - 2.x schedules overview: Schedules in Prefect 2

Great, thanks for sharing the effective config. You’re on Prefect 2.20 and your values look like the 2.x defaults, which helps. Why this can still take >20s - The scheduler isn’t just “a simple query.” Each loop: - Reads deployments/schedules to compute upcoming runs across a time horizon - Inserts new flow runs (batched) while avoiding duplicates - Repeats for “recently updated deployments” in a fast loop (defaults to 5s) - If tables are large, poorly indexed, bloated, or locked, these steps can stall. Even without high run volume, a wide scheduling horizon + many deployments/schedules or minute/second-level schedules can cause heavy work. - Your

PREFECT_API_DATABASE_TIMEOUT=10.0

means any one DB operation exceeding 10s will time out at the client. That matches the asyncpg TimeoutError you’re seeing. The 20s loop time includes retries/await plus the rest of the loop. Most actionable next steps Start with visibility so you fix the real cause rather than just raising timeouts. 1) Identify the slow/blocked queries - Temporarily enable slow query logging on RDS (e.g., set log_min_duration_statement=500ms) or use Performance Insights to see the top statements during scheduler spikes. - Check for locks/contention: - Query pg_stat_activity for waiting queries:

SELECT pid, wait_event_type, wait_event, state, query
    FROM pg_stat_activity
    WHERE datname = current_database()
    ORDER BY state, wait_event_type;

- If you can enable pg_stat_statements, look for the top queries by total time. 2) Reduce scheduler work per loop (2.x env vars) - These are 2.x names (prefix PREFECT_API_…). Try these first:

# Slow the “recent deployments” loop (default is 5s)
  PREFECT_API_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS=30

  # Reduce per-loop work and batching
  PREFECT_API_SERVICES_SCHEDULER_MAX_RUNS=50
  PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE=50
  PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE=250

  # Narrow the scheduling horizon (defaults: min=PT1H, max=P100D)
  PREFECT_API_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME=PT2H
  PREFECT_API_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME=P7D

- These changes reduce how much the scheduler must compute/insert each pass. 3) Loosen the client timeout while you tune - Your client timeout is currently 10s:

PREFECT_API_DATABASE_TIMEOUT=30

- Raising to 30–60 can stop spurious timeouts while you address the root cause. Don’t leave it too high long-term if queries are truly slow. 4) Right-size DB pooling and check total connections - Defaults are modest but multiply by the number of API/services pods. If you have several pods, connection counts can spike.

PREFECT_API_DATABASE_POOL_SIZE=5        # or 10 if your DB can handle it
  PREFECT_API_DATABASE_MAX_OVERFLOW=10
  PREFECT_API_DATABASE_POOL_TIMEOUT=30
  PREFECT_API_DATABASE_POOL_RECYCLE=3600

- Keep an eye on RDS max connections and CPU/IO. If the DB is saturated, the scheduler will suffer. 5) Check schedules and data volume - Do you have any minute/second-level RRule schedules (e.g., MINUTELY, BYSECOND)? Those are known to be heavy. Prefer Cron/Interval schedules or reduce frequency. - If you have millions of old flow_runs/flow_run_states/logs, scheduling queries can degrade. Ensure autovacuum/analyze are effective; consider manual VACUUM (FULL only with downtime) and REINDEX if bloat is high. 6) Optional: surface queries from Prefect - For a short diagnostic window, you can set:

PREFECT_API_DATABASE_ECHO=true

This logs SQL from the API process. It’s very verbose; use briefly in non-peak periods. What I’d do in your shoes - Immediately bump

PREFECT_API_DATABASE_TIMEOUT

to 30, and set: ``` PREFECT_API_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS=30 PREFECT_API_SERVICES_SCHEDULER_MAX_RUNS=50 PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE=50 PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE=250

PREFECT_API_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME=PT2H PREFECT_API_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME=P7D ``` - In parallel, enable RDS slow query logging or use Performance Insights to identify the exact hot statements and whether you’re hitting locks or table scans. - If you do have minute/second-level RRules, rewrite or reduce their frequency. References - 2.x settings: Prefect 2 settings - Known report on scheduler heaviness with RRule: Scheduler CPU spikes with RRule schedules If you can share roughly: - Number of deployments and schedule patterns - Approx row counts in flow_runs/flow_run_states - Number of API/services pods I can suggest more tailored values and help interpret any slow query data you find.

Thanks — that size on

flow_run_state

/

task_run_state

can definitely slow the scheduler. What query is “stuck”? - From the log snippet alone, we can’t see the exact SQL — Prefect 2.x does not log SQL by default. The asyncpg TimeoutError just tells us a DB call exceeded your

PREFECT_API_DATABASE_TIMEOUT

(10s in your config). - To identify the exact statement, use Postgres visibility: - See currently running/waiting queries:

SELECT pid,
           now() - query_start AS runtime,
           state,
           wait_event_type,
           wait_event,
           application_name,
           query
    FROM pg_stat_activity
    WHERE datname = current_database()
      AND state <> 'idle'
    ORDER BY query_start;

- If you can, enable pg_stat_statements and get the top time-consuming statements:

-- enable once in your DB/parameter group:
    -- CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

    SELECT total_exec_time, mean_exec_time, calls, query
    FROM pg_stat_statements
    ORDER BY total_exec_time DESC
    LIMIT 20;

- RDS Performance Insights (if enabled) will also show you the exact SQL and wait events. Optional: label Prefect connections so they’re easy to spot - Add an application name to your DSN so queries show up labeled: - Append

?application_name=prefect-api

to

PREFECT_API_DATABASE_CONNECTION_URL

- After rolling the pod, look for

application_name = 'prefect-api'

in

pg_stat_activity

. What the scheduler typically does - Reads deployments with active schedules - Computes upcoming run times (in Python) - Inserts new rows into

flow_runs

and corresponding states in batches - Checks for duplicates/limits If the state/run tables are very large and bloated or missing stats, even simple existence checks and inserts can slow down. Safe ways to reduce table size quickly 1) Delete old flow runs (cascades states and task runs) - In 2.x, FK constraints cascade states and task runs when you delete flow runs. If you’re comfortable purging old history, this is the fastest way to shrink

*_state

tables. - Example: delete completed flow runs older than 30 days. Adjust filters to your needs and test on a small window first.

-- recommended: create an index if not present to help this query:
  -- CREATE INDEX IF NOT EXISTS idx_flow_runs_end_time ON flow_runs (end_time);

  DELETE FROM flow_runs
  WHERE end_time IS NOT NULL
    AND end_time < now() - interval '30 days';

This will cascade delete from

flow_run_states

,

task_runs

, and

task_run_states

via FKs. 2) Prune historical states older than N days while keeping the current state - If you want to keep runs but trim old state history:

-- Flow run states: keep the state referenced by flow_runs.state_id, delete older ones
  DELETE FROM flow_run_states s
  USING flow_runs r
  WHERE s.flow_run_id = r.id
    AND s.id <> r.state_id
    AND s.timestamp < now() - interval '30 days';

  -- Task run states: same idea
  DELETE FROM task_run_states s
  USING task_runs r
  WHERE s.task_run_id = r.id
    AND s.id <> r.state_id
    AND s.timestamp < now() - interval '30 days';

Notes: - Ensure

timestamp

exists in your schema (it does in 2.x). - Do this in batches to avoid long locks, e.g., add

LIMIT

with a CTE or slice by time ranges. 3) Maintenance after purge - Update stats and reclaim space:

VACUUM (ANALYZE) flow_runs;
  VACUUM (ANALYZE) flow_run_states;
  VACUUM (ANALYZE) task_runs;
  VACUUM (ANALYZE) task_run_states;

For significant bloat and if you can afford locks/downtime:

VACUUM FULL

on the biggest tables or

REINDEX TABLE ...

. Tune the scheduler/db to stop the bleeding - In 2.x (your case), try these environment variables: ``` # Slow the “recent deployments” loop to reduce thrash PREFECT_API_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS=30 # Reduce work per pass PREFECT_API_SERVICES_SCHEDULER_MAX_RUNS=50

PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE=50 PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE=250 # Narrow the scheduling window (defaults: min=PT1H, max=P100D) PREFECT_API_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME=PT2H PREFECT_API_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME=P7D # Give DB calls more breathing room while you clean up PREFECT_API_DATABASE_TIMEOUT=30 ``` - Also confirm you’re not running multiple API/services pods each with their own pool; total connections add up. Find and fix the exact slow plan - Once you capture the slow statements via

pg_stat_activity

or Performance Insights, run

EXPLAIN (ANALYZE, BUFFERS)

on them to see if you need additional indexes. Common helpful indexes: -

flow_runs(deployment_id)

-

flow_runs(end_time)

-

flow_run_states(flow_run_id, timestamp)

-

task_run_states(task_run_id, timestamp)

- If you share the slow query text, I can suggest the exact index. Can logs show the SQL directly? - Not by default. You can temporarily set

PREFECT_API_DATABASE_ECHO=true

to log SQL from the API process (very verbose — use briefly and preferably off-peak). If you’d like, post: - One or two of the longest-running queries you see in

pg_stat_activity

or Performance Insights - Approximate row counts of

flow_runs

,

task_runs

I’ll help interpret and propose precise indexes and a pruning plan.

Totally fair questions. Why flow_run_states/task_run_states get huge - Every run accumulates a history of states. One flow run might create several flow_run_states over its lifetime; each task run can create even more (PENDING → SCHEDULED → RUNNING → RETRYING → RUNNING → COMPLETED/FAILED/CANCELLED, etc.). Large DAGs multiply this quickly. - The “data” column being NULL is normal; Prefect 2 doesn’t persist results by default. Table size comes from: - many rows (state history) - indexes on those rows - dead tuples/bloat from frequent inserts/updates/deletes if autovacuum can’t keep up - TOAST for large JSON fields (e.g., state_details) when present - If tables have been busy for a long time, they can become “chubby” even if current live rows aren’t massive, due to bloat. You’ll only reclaim disk with VACUUM FULL/pg_repack after deletions. Quick visibility to confirm - Check live vs dead tuples and total size:

SELECT relname, n_live_tup, n_dead_tup
FROM pg_stat_user_tables
WHERE relname IN ('flow_run_states','task_run_states');

SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) AS total
FROM pg_catalog.pg_statio_user_tables
WHERE relname IN ('flow_run_states','task_run_states');

-- average states per run (sanity checks)
SELECT
  (SELECT COUNT(*) FROM flow_run_states) AS flow_state_rows,
  (SELECT COUNT(*) FROM flow_runs) AS flow_runs,
  ROUND((SELECT COUNT(*) FROM flow_run_states)::numeric / GREATEST((SELECT COUNT(*) FROM flow_runs),1), 2) AS avg_flow_states_per_run;

SELECT
  (SELECT COUNT(*) FROM task_run_states) AS task_state_rows,
  (SELECT COUNT(*) FROM task_runs) AS task_runs,
  ROUND((SELECT COUNT(*) FROM task_run_states)::numeric / GREATEST((SELECT COUNT(*) FROM task_runs),1), 2) AS avg_task_states_per_run;

If n_dead_tup is large, you’ll get immediate wins from pruning + vacuum. You want to delete cancelled tasks (not completed) Here are safe, targeted cleanup options for Prefect 2.x. Always test on a small time window first and take a backup/snapshot before large deletes. Option A: Delete cancelled task runs older than N days (cascades their states) - This removes the entire cancelled task run rows and their state history, but keeps other runs intact.

-- Example: older than 30 days; adjust as needed
-- Helpful index if missing: CREATE INDEX IF NOT EXISTS idx_task_runs_end_time ON task_runs (end_time);

WITH doomed AS (
  SELECT tr.id
  FROM task_runs tr
  JOIN task_run_states s ON s.id = tr.state_id
  WHERE s.type IN ('CANCELLED','CANCELLING')
    AND tr.end_time IS NOT NULL
    AND tr.end_time < now() - interval '30 days'
  LIMIT 10000
)
DELETE FROM task_runs tr
WHERE tr.id IN (SELECT id FROM doomed);

-- Repeat the DELETE until it affects 0 rows to process in batches.

Option B: Keep runs but prune old state history (big win for table size) - Keep the current state each run points to; delete older historical states for cancelled task runs only (or do this for all task runs if you want maximal reduction).

-- Prune task_run_states for CANCELLED runs, keeping the current state
DELETE FROM task_run_states s
USING task_runs r, task_run_states cur
WHERE s.task_run_id = r.id
  AND cur.id = r.state_id
  AND cur.type IN ('CANCELLED','CANCELLING')
  AND s.id <> r.state_id
  AND s.timestamp < now() - interval '30 days'
LIMIT 50000;  -- run in batches

- You can run a similar prune for flow_run_states if desired:

DELETE FROM flow_run_states s
USING flow_runs r, flow_run_states cur
WHERE s.flow_run_id = r.id
  AND cur.id = r.state_id
  AND cur.type IN ('CANCELLED','CANCELLING')
  AND s.id <> r.state_id
  AND s.timestamp < now() - interval '30 days'
LIMIT 50000;

After cleanup, reclaim space - Update stats so planners pick the right indexes:

VACUUM (ANALYZE) task_run_states;
VACUUM (ANALYZE) flow_run_states;
VACUUM (ANALYZE) task_runs;
VACUUM (ANALYZE) flow_runs;

- To truly shrink on-disk size, consider:

- VACUUM FULL task_run_states; VACUUM FULL flow_run_states; (requires locks/downtime) - or use pg_repack for online compaction Why “no open query” but scheduler times out? - The timeouts happen during scheduler spikes; you may have checked pg_stat_activity between spikes. Adding an application_name to your DSN helps you spot Prefect queries quickly: - Append

?application_name=prefect-api

to

PREFECT_API_DATABASE_CONNECTION_URL

- If you still see no active queries at the time of warnings, the bottleneck may be intermittent (autovacuum, IO contention) or the scheduler loop is delayed by something else. Enabling RDS Performance Insights or pg_stat_statements for a day will typically reveal the heavy statements. Two more tips that usually help - Narrow the scheduler horizon and reduce per-loop work (2.x env vars):

PREFECT_API_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS=30
PREFECT_API_SERVICES_SCHEDULER_MAX_RUNS=50
PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE=50
PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE=250
PREFECT_API_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME=PT2H
PREFECT_API_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME=P7D
PREFECT_API_DATABASE_TIMEOUT=30

- Ensure autovacuum keeps up. If n_dead_tup is high for states tables, consider increasing autovacuum aggressiveness for those tables or scheduling periodic manual VACUUM. If you can share: - Whether “9M” for flow_runs is rows or 9 MB - The avg states per run from the queries above - A sample slow statement from Performance Insights/pg_stat_statements I can recommend exact indexes and a tailored pruning cadence (and convert the deletes into a scheduled job).

Got it — and great catch on the table names. Please adapt the queries below to your exact schema (plural vs singular). I’ll show them in the canonical Prefect 2.x schema and note where to change. Your averages are the smoking gun - Avg flow states per run: 11,307 - Avg task states per run: 7,601 That’s extraordinarily high. Even without table bloat, that row volume will slow scheduling checks. Pruning historical states will help a lot, and we should later look for the cause of the state “storm” so it doesn’t recur. Prune state history: keep only the current/final state for terminal runs - Keep the state that each run currently points to (

r.state_id

) when the run is terminal (COMPLETED/FAILED/CANCELLED/CRASHED). Delete all other historical states for that run. - Do it in batches to avoid long locks. Run repeatedly until 0 rows deleted. Flow run states (canonical names; change to your singular names if needed)

-- Optional: add indexes to speed joins if missing
-- CREATE INDEX IF NOT EXISTS idx_flow_run_states_flow_run_id ON flow_run_states (flow_run_id);
-- CREATE INDEX IF NOT EXISTS idx_flow_runs_state_id ON flow_runs (state_id);

WITH doomed AS (
  SELECT s.id
  FROM flow_run_states s
  JOIN flow_runs r ON r.id = s.flow_run_id
  JOIN flow_run_states cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
)
DELETE FROM flow_run_states
WHERE id IN (SELECT id FROM doomed);

Task run states

-- Optional indexes
-- CREATE INDEX IF NOT EXISTS idx_task_run_states_task_run_id ON task_run_states (task_run_id);
-- CREATE INDEX IF NOT EXISTS idx_task_runs_state_id ON task_runs (state_id);

WITH doomed AS (
  SELECT s.id
  FROM task_run_states s
  JOIN task_runs r ON r.id = s.task_run_id
  JOIN task_run_states cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
)
DELETE FROM task_run_states
WHERE id IN (SELECT id FROM doomed);

Notes - Replace `flow_run_states`/`task_run_states` with your actual table names (`flow_run_state`/`task_run_state`) if that’s what your schema uses. - You can add an age filter if you want to be extra safe: -

AND s.timestamp < now() - interval '7 days'

- After each batch, re-run until 0 rows deleted. Follow-up maintenance

VACUUM (ANALYZE) flow_run_states;
VACUUM (ANALYZE) task_run_states;
-- For significant space reclaim (requires locks/downtime):
-- VACUUM FULL flow_run_states;
-- VACUUM FULL task_run_states;

If downtime is hard, consider pg_repack for online compaction. Quick sanity checks before/after

-- Find the worst offenders (runs with the most states)
SELECT flow_run_id, COUNT(*) AS n
FROM flow_run_states
GROUP BY flow_run_id
ORDER BY n DESC
LIMIT 20;

SELECT task_run_id, COUNT(*) AS n
FROM task_run_states
GROUP BY task_run_id
ORDER BY n DESC
LIMIT 20;

Why did states balloon? Averages in the thousands suggest something is repeatedly flipping states. Common causes: - High retry counts with short delays or tasks that keep re-entering RUNNING/SCHEDULED - Processes repeatedly re-attaching to the same runs after restarts (workers crashing/restarting) - Custom code or triggers that set states frequently - Services like “late runs” thrashing on problematic runs If you can share a couple of the top offending run IDs and their state type distribution, we can diagnose:

-- Distribution of state types for a specific run
SELECT type, COUNT(*) 
FROM flow_run_states 
WHERE flow_run_id = '<uuid>'
GROUP BY type
ORDER BY COUNT(*) DESC;

To prevent recurrence - Keep a periodic prune job (weekly/daily) that keeps only the current state for terminal runs (queries above without LIMIT). - Review retry policies and ensure they’re not excessively aggressive. - Check for worker restarts/health that could cause repeated transitions.

- Narrow scheduler work temporarily while you cleanup:

PREFECT_API_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS=30
PREFECT_API_SERVICES_SCHEDULER_MAX_RUNS=50
PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE=50
PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE=250
PREFECT_API_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME=PT2H
PREFECT_API_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME=P7D
PREFECT_API_DATABASE_TIMEOUT=30

If you want, paste one example run’s state-type breakdown and your retry config; I’ll help pinpoint the pattern and suggest guardrails so the states table doesn’t balloon again.

Whoa — 655k CANCELLING states on a single flow run is the smoking gun. That’s not normal churn; something is repeatedly re-applying CANCELLING. DB load can exacerbate it (the finalization to CANCELLED may never complete), but the core issue is a loop that keeps writing CANCELLING. What I’d do now 1) Force-finalize the top offenders to stop the bleed - For each of those flow run IDs, try to force them to CANCELLED so Prefect stops re-asserting CANCELLING:

prefect flow-run cancel dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5
prefect flow-run cancel d8c7c0cb-2909-4eb0-b3f1-fcbfb6a71079
...

- Then confirm the current state:

SELECT r.id, s.type, s.timestamp
FROM flow_runs r
JOIN flow_run_state s ON s.id = r.state_id
WHERE r.id IN ('dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5','d8c7c0cb-2909-4eb0-b3f1-fcbfb6a71079');

2) Prune historical states, keeping only the final state - Use your singular table names. Run in batches until no more rows are deleted. This keeps only the final state for terminal runs.

-- Flow runs: keep only the current terminal state
WITH doomed AS (
  SELECT s.id
  FROM flow_run_state s
  JOIN flow_runs r ON r.id = s.flow_run_id
  JOIN flow_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
)
DELETE FROM flow_run_state
WHERE id IN (SELECT id FROM doomed);

-- Task runs: same pattern
WITH doomed AS (
  SELECT s.id
  FROM task_run_state s
  JOIN task_runs r ON r.id = s.task_run_id
  JOIN task_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
)
DELETE FROM task_run_state
WHERE id IN (SELECT id FROM doomed);

- Optional safety: also add

AND s.timestamp < now() - interval '7 days'

if you want to keep the most recent history. - Maintenance after pruning:

VACUUM (ANALYZE) flow_run_state;
VACUUM (ANALYZE) task_run_state;
-- For real space reclaim (locks/downtime): VACUUM FULL flow_run_state; VACUUM FULL task_run_state;

3) Identify who/what is re-asserting CANCELLING - Look for repeated cancellation requests in the state messages/details:

SELECT message, COUNT(*)
FROM flow_run_state
WHERE flow_run_id = 'dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5'
  AND type = 'CANCELLING'
GROUP BY message
ORDER BY COUNT(*) DESC
LIMIT 10;

SELECT state_details, COUNT(*)
FROM flow_run_state
WHERE flow_run_id = 'dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5'
  AND type = 'CANCELLING'
GROUP BY state_details
ORDER BY COUNT(*) DESC
LIMIT 10;

- Common culprits: - An external script repeatedly calling the cancel API for the same run - The cancellation cleanup service re-touching runs that never finalize because workers are offline or failing to report CANCELLED - Worker restarts or crashes causing repeated “cancel requested” handling - Check worker logs for these run IDs and the server logs around when CANCELLING spikes. 4) Temporarily reduce/disable the cancellation churn while you clean - If available in your 2.20 build, you can slow or temporarily disable cancellation cleanup to stop the flood while you prune. Look for an env like:

PREFECT_API_SERVICES_CANCELLATION_CLEANUP_ENABLED=false

- If that setting isn’t present in your 2.20 image, slow the scheduler loops and raise the DB timeout while you clean:

PREFECT_API_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS=30
PREFECT_API_SERVICES_SCHEDULER_MAX_RUNS=50
PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE=50
PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE=250
PREFECT_API_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME=PT2H
PREFECT_API_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME=P7D
PREFECT_API_DATABASE_TIMEOUT=30

5) Make it observable - Tag Prefect DB connections so you can spot them: - Append

?application_name=prefect-api

to

PREFECT_API_DATABASE_CONNECTION_URL

- If needed briefly, enable SQL echo to correlate behavior:

PREFECT_API_DATABASE_ECHO=true

Use sparingly (very verbose). Your numbers confirm the plan - Avg 11k flow states/run and 7.6k task states/run = extreme churn. - Dead tuples near zero → not (just) bloat; these are live rows. Pruning is the right fix. - Once you force-finalize the worst runs, pruning will cut the states table dramatically. Then VACUUM/pg_repack to reclaim space. If you share: - The message/state_details sample from the CANCELLING rows - Whether those runs’ current state is CANCELLING or CANCELLED after

prefect flow-run cancel

I can help pinpoint the source and suggest a guardrail so this doesn’t recur.

That message pinpoints it: these are child runs being repeatedly put into CANCELLING because “The parent flow run was cancelled.” So the cancellation enforcer keeps reasserting CANCELLING on the children and never finalizes them, creating hundreds of thousands of new state rows. DB load can absolutely exacerbate this: if finalization (moving to CANCELLED) can’t complete fast enough, the cleanup/enforcer will keep touching the same runs and append more CANCELLING states. Here’s a focused plan to stop the churn, clean up safely, and prevent recurrence. Immediate containment 1) Force-finalize the worst offenders - Try to move them from CANCELLING to CANCELLED via CLI so the enforcer stops touching them:

prefect flow-run cancel dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5
prefect flow-run cancel d8c7c0cb-2909-4eb0-b3f1-fcbfb6a71079
prefect flow-run cancel 1c57c018-8154-4f5b-b0d5-c53e325207ff

- Verify:

SELECT r.id, s.type, s.timestamp
FROM flow_runs r
JOIN flow_run_state s ON s.id = r.state_id
WHERE r.id IN ('dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5','d8c7c0cb-2909-4eb0-b3f1-fcbfb6a71079','1c57c018-8154-4f5b-b0d5-c53e325207ff');

2) Optional: slow cancellation enforcement temporarily - If available in your 2.20 image, you can slow or pause cancellation cleanup while you clean to stop re-assertions. If not present, at least raise

PREFECT_API_DATABASE_TIMEOUT

and reduce scheduler work temporarily as shared earlier to reduce the rate of reassertion. - Ref: Prefect 2 settings Prune state history (Option B) — tuned for your singular table names A) Terminal runs: keep only the current terminal state; delete all older states

WITH doomed AS (
  SELECT s.id
  FROM flow_run_state s
  JOIN flow_runs r ON r.id = s.flow_run_id
  JOIN flow_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
)
DELETE FROM flow_run_state
WHERE id IN (SELECT id FROM doomed);

WITH doomed AS (
  SELECT s.id
  FROM task_run_state s
  JOIN task_runs r ON r.id = s.task_run_id
  JOIN task_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
)
DELETE FROM task_run_state
WHERE id IN (SELECT id FROM doomed);

B) Runs still stuck in CANCELLING: keep only the latest CANCELLING state - First, ensure flow_runs.state_id points to the latest CANCELLING state for consistency:

WITH latest AS (
  SELECT DISTINCT ON (s.flow_run_id) s.id, s.flow_run_id, s.timestamp
  FROM flow_run_state s
  WHERE s.type = 'CANCELLING'
  ORDER BY s.flow_run_id, s.timestamp DESC
)
UPDATE flow_runs r
SET state_id = latest.id
FROM latest
JOIN flow_run_state cur ON cur.id = r.state_id
WHERE r.id = latest.flow_run_id
  AND cur.type = 'CANCELLING'
  AND r.state_id <> latest.id;

- Then delete older CANCELLING rows:

WITH latest AS (
  SELECT flow_run_id, MAX(timestamp) AS ts
  FROM flow_run_state
  WHERE type = 'CANCELLING'
  GROUP BY flow_run_id
)
DELETE FROM flow_run_state s
USING latest l
JOIN flow_run_state cur ON cur.id = (SELECT state_id FROM flow_runs WHERE id = l.flow_run_id)
WHERE s.type = 'CANCELLING'
  AND s.flow_run_id = l.flow_run_id
  AND cur.type = 'CANCELLING'
  AND s.timestamp < l.ts
LIMIT 50000;

- Run these in batches until 0 rows deleted. Then:

VACUUM (ANALYZE) flow_run_state;
VACUUM (ANALYZE) task_run_state;
-- For disk reclaim (requires locks/downtime): VACUUM FULL flow_run_state; VACUUM FULL task_run_state;

Find the parent and the cascade pattern - For one offender, identify its parent and siblings — verify the parent is CANCELLED and children are the ones churning: ``` -- Find the parent flow run of a child subflow SELECT parent.id AS parent_flow_run_id, child.id AS child_flow_run_id FROM flow_runs child

JOIN task_runs t ON child.parent_task_run_id = t.id JOIN flow_runs parent ON t.flow_run_id = parent.id WHERE child.id = 'dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5'; -- Are siblings also stuck cancelling? SELECT child.id, cur.type, cur.timestamp FROM flow_runs child JOIN task_runs t ON child.parent_task_run_id = t.id JOIN flow_run_state cur ON cur.id = child.state_id WHERE t.flow_run_id = '<parent_flow_run_id>' ORDER BY cur.type, cur.timestamp DESC LIMIT 50;

- If you see many children in CANCELLING, finalize them (CLI cancel) and then prune.

Prevent recurrence
- Ensure workers/infrastructure actually terminate runs on cancel. If you’re using Docker/K8s/Dask/Ray, verify your cancellation behavior kills child processes; otherwise the server will keep re-requesting cancellation.
- Add run-level timeouts and reasonable retry policies to avoid long-running stuck runs.
- Keep a periodic prune job (daily/weekly) that:
  - For terminal runs: keeps only current state
  - For runs stuck in CANCELLING for > N minutes: keeps only the latest CANCELLING state or proactively finalizes them to CANCELLED
- While you clean up, reduce scheduler pressure and bump DB timeout:

PREFECT_API_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS=30 PREFECT_API_SERVICES_SCHEDULER_MAX_RUNS=50 PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE=50 PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE=250 PREFECT_API_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME=PT2H PREFECT_API_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME=P7D PREFECT_API_DATABASE_TIMEOUT=30 ``` Refs - Settings: Prefect 2 settings - Related scheduler heaviness report: RRule schedules causing scheduler CPU and long loops If you can share whether those children finalize to CANCELLED after the CLI cancel, we can confirm the loop is broken and then aggressively prune the historical CANCELLING rows.

You’re right:

prefect flow-run cancel

only sets the run to CANCELLING. Normally, a worker or the server’s cleanup service will finalize it to CANCELLED. Since yours are stuck, the fastest way is to force the final state. Two ways to force-finalize stuck runs to CANCELLED A) Use the REST API with force=true - Adjust for your API URL shape. If your

PREFECT_API_URL

already ends with

/api

, use it directly; otherwise add

/api

. If you use auth, add the Authorization header.

curl -X POST "$PREFECT_API_URL/flow_runs/dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5/set_state?force=true" \
  -H "Content-Type: application/json" \
  -d '{"state": {"type":"CANCELLED","name":"Cancelled","message":"Forced cancel"}}'

- Repeat for each offender. Then verify:

SELECT r.id, s.type, s.timestamp
FROM flow_runs r
JOIN flow_run_state s ON s.id = r.state_id
WHERE r.id IN ('dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5','d8c7c0cb-2909-4eb0-b3f1-fcbfb6a71079');

B) Use a tiny Python script (Prefect 2.x)

import asyncio
from prefect.client.orchestration import PrefectClient
from prefect.states import Cancelled

RUN_IDS = [
    "dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5",
    "d8c7c0cb-2909-4eb0-b3f1-fcbfb6a71079",
    # ...
]

async def main():
    async with PrefectClient() as client:
        for rid in RUN_IDS:
            state = Cancelled(name="Forced cancel")
            await client.set_flow_run_state(rid, state=state, force=True)
            print("Forced CANCELLED:", rid)

asyncio.run(main())

- This bypasses orchestration guardrails and sets the final state. Then prune the storm of CANCELLING states (Option B) - Keep only the current state for terminal runs and delete older history. Since your schema uses singular names, use these:

WITH doomed AS (
  SELECT s.id
  FROM flow_run_state s
  JOIN flow_runs r ON r.id = s.flow_run_id
  JOIN flow_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
)
DELETE FROM flow_run_state
WHERE id IN (SELECT id FROM doomed);

WITH doomed AS (
  SELECT s.id
  FROM task_run_state s
  JOIN task_runs r ON r.id = s.task_run_id
  JOIN task_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
)
DELETE FROM task_run_state
WHERE id IN (SELECT id FROM doomed);

- Run in batches until 0 rows; then:

VACUUM (ANALYZE) flow_run_state;
VACUUM (ANALYZE) task_run_state;
-- For real disk reclaim (locks/downtime):
-- VACUUM FULL flow_run_state; VACUUM FULL task_run_state;

If a run is still in CANCELLING after the forced API call - Confirm the HTTP response was 200 and that you used

force=true

. - Make sure you hit the correct base URL (some setups have

PREFECT_API_URL

already including

/api

). - If it still won’t stick, we can do a surgical DB remediation: insert a new CANCELLED state row and point

flow_runs.state_id

to it. I can give you that SQL if needed; the API method is preferred. Reduce the chance of recurrence - These children are getting repeatedly set to CANCELLING because “The parent flow run was cancelled.” Once you force them to CANCELLED, the loop should stop. - Ensure cancellations actually terminate child work on your infra (Docker/K8s/etc.); otherwise the server keeps reasserting CANCELLING. - Consider a periodic job that: - For terminal runs, keeps only the current state (queries above without LIMIT) - For runs stuck in CANCELLING for > N minutes, forcibly sets CANCELLED via API If you want, share the exact API URL you’re using and the response you get from the force call; I can sanity-check the endpoint and payload.

That error is because in Prefect 2.x,

PrefectClient

requires an

api

argument. The easiest fix is to use

get_client()

which reads

PREFECT_API_URL

from your env. Try one of these two options. Option A: use get_client() (recommended)

import asyncio
from prefect.client.orchestration import get_client
from prefect.states import Cancelled

RUN_IDS = [
    "dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5",
    "d8c7c0cb-2909-4eb0-b3f1-fcbfb6a71079",
    # ...
]

async def main():
    async with get_client() as client:  # picks up PREFECT_API_URL
        for rid in RUN_IDS:
            state = Cancelled(name="Forced cancel")
            resp = await client.set_flow_run_state(rid, state=state, force=True)
            print("Forced CANCELLED:", rid, getattr(resp, "status", resp))

asyncio.run(main())

Option B: pass the API URL explicitly

import os, asyncio
from prefect.client.orchestration import PrefectClient
from prefect.states import Cancelled

API = os.environ["PREFECT_API_URL"]  # e.g. http://<host>:4200/api
RUN_IDS = ["dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5", "d8c7c0cb-2909-4eb0-b3f1-fcbfb6a71079"]

async def main():
    async with PrefectClient(api=API) as client:
        for rid in RUN_IDS:
            state = Cancelled(name="Forced cancel")
            resp = await client.set_flow_run_state(rid, state=state, force=True)
            print("Forced CANCELLED:", rid, getattr(resp, "status", resp))

asyncio.run(main())

If you prefer cURL, you can force via the REST API as well (adjust base URL and auth as needed):

curl -X POST "$PREFECT_API_URL/flow_runs/dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5/set_state?force=true" \
  -H "Content-Type: application/json" \
  -d '{"state": {"type":"CANCELLED","name":"Cancelled","message":"Forced cancel"}}'

If the forced set still doesn’t stick or returns a rejection, tell me the response you get and we’ll either adjust the request or do a surgical DB fix: - Insert a CANCELLED state row, then point the run to it:

-- Replace with your singular table names if needed
WITH new_state AS (
  INSERT INTO flow_run_state (flow_run_id, type, name, message, timestamp)
  VALUES ('dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5', 'CANCELLED', 'Cancelled', 'Forced cancel', now())
  RETURNING id
)
UPDATE flow_runs r
SET state_id = new_state.id
FROM new_state
WHERE r.id = 'dd01ef7c-7ba6-4bce-9fb1-c1612bfe5ff5';

Then run your pruning queries to drop the hundreds of thousands of old CANCELLING rows and VACUUM. Once you confirm the forced CANCELLED worked, I’d: - Prune historical states (keep only the current state for terminal runs) - Consider a small periodic job to do that pruning going forward - Double-check your infra cancellation actually terminates child work, so the server doesn’t keep reasserting CANCELLING on children of a cancelled parent If anything in the snippets above throws or returns a REJECT/WAI status, paste it here and I’ll adjust the approach.

Nice — glad the force worked. About “Forced Cancel” vs CANCELLED - Prefect states have a type and a display name. You set a Cancelled state with name “Forced cancel”, so: - type = CANCELLED - name = Forced cancel (what the UI shows) - You do not need to change anything if your scripts/SQL filter by state type. If you’re checking the name, switch to type. - Good pattern:

WHERE type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')

Should you fix the ~100 runs with ~10k states? - 10k states/run is still very high. It’s worth investigating, but you don’t have to stop everything for them right now. - First, prune history so they don’t waste space: - The “keep only the current state for terminal runs” query will clean these too if they’re terminal. - For non-terminal runs stuck in CANCELLING, keep only the latest CANCELLING state. Here are the cleanup queries using your singular table names. Prune terminal runs (keep only current terminal state)

WITH doomed AS (
  SELECT s.id
  FROM flow_run_state s
  JOIN flow_runs r ON r.id = s.flow_run_id
  JOIN flow_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
)
DELETE FROM flow_run_state
WHERE id IN (SELECT id FROM doomed);

WITH doomed AS (
  SELECT s.id
  FROM task_run_state s
  JOIN task_runs r ON r.id = s.task_run_id
  JOIN task_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
)
DELETE FROM task_run_state
WHERE id IN (SELECT id FROM doomed);

Run repeatedly until 0 rows deleted; then:

VACUUM (ANALYZE) flow_run_state;
VACUUM (ANALYZE) task_run_state;

For runs still in CANCELLING: keep only the latest CANCELLING state

-- Ensure flow_runs.state_id points to the latest CANCELLING
WITH latest AS (
  SELECT DISTINCT ON (s.flow_run_id) s.id, s.flow_run_id, s.timestamp
  FROM flow_run_state s
  WHERE s.type = 'CANCELLING'
  ORDER BY s.flow_run_id, s.timestamp DESC
)
UPDATE flow_runs r
SET state_id = latest.id
FROM latest
JOIN flow_run_state cur ON cur.id = r.state_id
WHERE r.id = latest.flow_run_id
  AND cur.type = 'CANCELLING'
  AND r.state_id <> latest.id;

-- Delete older CANCELLING rows
WITH latest AS (
  SELECT flow_run_id, MAX(timestamp) AS ts
  FROM flow_run_state
  WHERE type = 'CANCELLING'
  GROUP BY flow_run_id
)
DELETE FROM flow_run_state s
USING latest l
JOIN flow_run_state cur ON cur.id = (SELECT state_id FROM flow_runs WHERE id = l.flow_run_id)
WHERE s.type = 'CANCELLING'
  AND s.flow_run_id = l.flow_run_id
  AND cur.type = 'CANCELLING'
  AND s.timestamp < l.ts
LIMIT 50000;

Diagnose why those 10k runs ballooned - Check the state-type distribution on a few of them. If it’s mostly CANCELLING/RETRYING/RUNNING flips, there’s a behavioral loop to address.

SELECT type, COUNT(*)
FROM flow_run_state
WHERE flow_run_id = '<run_id_with_10k_states>'
GROUP BY type
ORDER BY COUNT(*) DESC;

- Common causes: - Parent cancellation cascading to many children that never finalize, repeatedly reasserting CANCELLING - Very aggressive retries or short retry delays causing rapid churn - Infrastructure not actually terminating work on cancel (server keeps re-requesting cancel) - Automation repeatedly calling cancel on the same runs Prevention - Keep the pruning job as a periodic maintenance task. - Ensure cancellations on your infra truly terminate processes for subflows/tasks. - Tame retry policies and add sensible timeouts to flows/tasks to avoid long “stuck” runs. Once your cleanup is done and things stabilize, you can dial back any temporary scheduler/DB tweaks you set earlier. If you want, share one 10k run’s state distribution and I’ll help pinpoint the cause and suggest exact guardrails.

You’re right: big one-shot deletes get slower as the planner builds large temp sets and you hold long locks. Two things will help a lot: - Make the delete selection cheap (use indexed joins and avoid large materializations) - Commit between batches Prep: add the cheap join indexes if they’re missing

-- For flow states
CREATE INDEX IF NOT EXISTS idx_flow_run_state_flow_run_id ON flow_run_state (flow_run_id);
CREATE INDEX IF NOT EXISTS idx_flow_runs_state_id ON flow_runs (state_id);

-- For task states
CREATE INDEX IF NOT EXISTS idx_task_run_state_task_run_id ON task_run_state (task_run_id);
CREATE INDEX IF NOT EXISTS idx_task_runs_state_id ON task_runs (state_id);

ANALYZE flow_run_state; ANALYZE flow_runs;
ANALYZE task_run_state; ANALYZE task_runs;

Fast chunked delete pattern (no CTE, use ctid and commit each batch) - This pattern avoids materializing a huge CTE and lets you run many small commits. Run these repeatedly until 0 rows are deleted each time. Adjust chunk size to what your DB tolerates (start 20k–50k). Flow run states (keep only current state for terminal runs)

DELETE FROM flow_run_state s
USING (
  SELECT s.ctid
  FROM flow_run_state s
  JOIN flow_runs r ON r.id = s.flow_run_id
  JOIN flow_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
) d
WHERE s.ctid = d.ctid;

Task run states

DELETE FROM task_run_state s
USING (
  SELECT s.ctid
  FROM task_run_state s
  JOIN task_runs r ON r.id = s.task_run_id
  JOIN task_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
) d
WHERE s.ctid = d.ctid;

How to loop with commits - Easiest is a tiny client script that repeats the delete until it returns 0 rows, committing each batch. Example with psycopg (commit per batch)

import os, time, psycopg2

dsn = os.environ["PREFECT_DB_DSN"]  # e.g. <postgresql://user:pass@host:5432/db>
chunk = 50000

delete_flow_sql = f"""
DELETE FROM flow_run_state s
USING (
  SELECT s.ctid
  FROM flow_run_state s
  JOIN flow_runs r ON r.id = s.flow_run_id
  JOIN flow_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT {chunk}
) d
WHERE s.ctid = d.ctid;
"""

delete_task_sql = f"""
DELETE FROM task_run_state s
USING (
  SELECT s.ctid
  FROM task_run_state s
  JOIN task_runs r ON r.id = s.task_run_id
  JOIN task_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT {chunk}
) d
WHERE s.ctid = d.ctid;
"""

with psycopg2.connect(dsn) as conn:
    conn.autocommit = False
    with conn.cursor() as cur:
        while True:
            cur.execute(delete_flow_sql)
            deleted = cur.rowcount
            conn.commit()
            print("flow_run_state deleted:", deleted)
            if deleted == 0:
                break
            time.sleep(0.2)

        while True:
            cur.execute(delete_task_sql)
            deleted = cur.rowcount
            conn.commit()
            print("task_run_state deleted:", deleted)
            if deleted == 0:
                break
            time.sleep(0.2)

Alternative: temp table of terminal runs for a faster join - If you have many non-terminal runs, precomputing terminal runs can make deletes cheaper: ``` CREATE TEMP TABLE term_flow_runs AS SELECT r.id AS flow_run_id, r.state_id AS keep_state_id FROM flow_runs r JOIN flow_run_state cur ON cur.id = r.state_id WHERE cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED'); CREATE INDEX ON term_flow_runs (flow_run_id); CREATE INDEX ON term_flow_runs (keep_state_id); ANALYZE term_flow_runs; -- Then chunked delete becomes: DELETE FROM flow_run_state s USING (

SELECT s.ctid FROM flow_run_state s JOIN term_flow_runs t ON t.flow_run_id = s.flow_run_id WHERE s.id <> t.keep_state_id LIMIT 50000 ) d WHERE s.ctid = d.ctid;

- Do the same for task runs if needed.

Session tweaks while deleting
- Run during off-peak if you can.
- Consider temporarily raising statement_timeout for your session or disabling it:

SET LOCAL statement_timeout = '0'; -- no timeout for the maintenance session SET LOCAL lock_timeout = '2s'; -- don’t wait forever on conflicting locks SET LOCAL application_name = 'prefect-prune';

After the big delete

VACUUM (ANALYZE) flow_run_state; VACUUM (ANALYZE) task_run_state; -- To reclaim disk (requires locks/downtime): VACUUM FULL flow_run_state; VACUUM FULL task_run_state; -- Or use pg_repack for online compaction if available ``` How long will ~4M rows take? - If you’re getting ~50k/min with small locks/commits, you’re looking at around 80 minutes for 4M — that’s reasonable. Larger batches can backfire due to memory and longer locks. The chunk+commit approach typically gives the best wall-clock with the least impact on other workload. If you hit any lock waits or see the delete still scanning too much, share EXPLAIN for the chunked delete and we’ll tune indexes further.

Great question — both patterns work, but they serve different purposes: - Using id: stable, simple, relies on an index on id. Good if your join is selective and the PK/index is efficient. - Using ctid: Postgres-only “row pointer” that’s very fast for chunked deletes because we select the exact rows to delete and match them by ctid in the same statement. Don’t store ctids; only use them inside the same statement. Given your volume, use the ctid pattern for speed. Here are consistent, singular-table snippets you can run in a loop (commit between batches). Prep (once)

CREATE INDEX IF NOT EXISTS idx_flow_run_state_flow_run_id ON flow_run_state (flow_run_id);
CREATE INDEX IF NOT EXISTS idx_flow_runs_state_id ON flow_runs (state_id);

CREATE INDEX IF NOT EXISTS idx_task_run_state_task_run_id ON task_run_state (task_run_id);
CREATE INDEX IF NOT EXISTS idx_task_runs_state_id ON task_runs (state_id);

ANALYZE flow_run_state; ANALYZE flow_runs;
ANALYZE task_run_state; ANALYZE task_runs;

Chunked delete using ctid (terminal runs: keep only current state)

DELETE FROM flow_run_state s
USING (
  SELECT s.ctid
  FROM flow_run_state s
  JOIN flow_runs r ON r.id = s.flow_run_id
  JOIN flow_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
) pick
WHERE s.ctid = pick.ctid;

DELETE FROM task_run_state s
USING (
  SELECT s.ctid
  FROM task_run_state s
  JOIN task_runs r ON r.id = s.task_run_id
  JOIN task_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
) pick
WHERE s.ctid = pick.ctid;

If you prefer the id variant (slightly simpler, may be a bit slower):

DELETE FROM flow_run_state s
USING (
  SELECT s.id
  FROM flow_run_state s
  JOIN flow_runs r ON r.id = s.flow_run_id
  JOIN flow_run_state cur ON cur.id = r.state_id
  WHERE r.state_id IS NOT NULL
    AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
    AND s.id <> r.state_id
  LIMIT 50000
) pick
WHERE s.id = pick.id;

Pure SQL loop (runs on the server; stops when a batch deletes 0 rows)

DO $$
DECLARE
  r BIGINT;
BEGIN
  LOOP
    DELETE FROM flow_run_state s
    USING (
      SELECT s.ctid
      FROM flow_run_state s
      JOIN flow_runs r ON r.id = s.flow_run_id
      JOIN flow_run_state cur ON cur.id = r.state_id
      WHERE r.state_id IS NOT NULL
        AND cur.type IN ('COMPLETED','FAILED','CANCELLED','CRASHED')
        AND s.id <> r.state_id
      LIMIT 50000
    ) pick
    WHERE s.ctid = pick.ctid;

    GET DIAGNOSTICS r = ROW_COUNT;
    RAISE NOTICE 'flow_run_state deleted %', r;
    IF r = 0 THEN EXIT; END IF;
  END LOOP;
END $$;

Session tips while running batches

SET LOCAL application_name = 'prefect-prune';
SET LOCAL lock_timeout = '2s';
SET LOCAL statement_timeout = '0';  -- just for this maintenance session

After you finish

VACUUM (ANALYZE) flow_run_state;
VACUUM (ANALYZE) task_run_state;
-- To reclaim disk (locks/downtime):
-- VACUUM FULL flow_run_state; VACUUM FULL task_run_state;

TL;DR: Use the ctid-based chunked delete inside a loop (or the DO $$ loop above). It’s normal that each 50k batch takes ~1 minute on large tables; the loop with commits will still finish faster overall and with less lock impact than giant single-shot deletes. Adjust the LIMIT up/down to hit your sweet spot.

You didn’t do anything wrong. VACUUM (even with ANALYZE) does not shrink on-disk files; it only marks space for reuse inside the table. That’s why DBeaver still shows ~9 GB. To actually reduce the table file size you have to rewrite the table. What to do next 1) Verify where the size is (heap vs indexes vs toast)

SELECT relname,
  pg_size_pretty(pg_relation_size(oid))    AS table_size,
  pg_size_pretty(pg_indexes_size(oid))     AS index_size,
  pg_size_pretty(pg_total_relation_size(oid)) AS total_size
FROM pg_class
WHERE relname IN ('flow_run_state','task_run_state');

2) If you need the physical files to shrink: - Option A (simple, requires downtime/locks): VACUUM FULL rewrites and shrinks

VACUUM FULL flow_run_state;
VACUUM FULL task_run_state;

This takes an exclusive lock per table. Do it in a maintenance window (stop Prefect API/services and workers while it runs). - Option B (online): pg_repack - Rewrites tables and indexes online with minimal blocking. Requires installing/using pg_repack client and extension. - Docs: https://reorg.github.io/pg_repack/ - Good when you can’t take locks but want disk back. - Option C (manual rewrite): create a new table, copy rows, swap, drop old - More operational work (constraints, indexes, FKs), but also shrinks files. 3) Don’t forget indexes - Indexes can be a big chunk of that 9 GB. If you don’t do VACUUM FULL or pg_repack, you can at least shrink indexes:

REINDEX TABLE CONCURRENTLY flow_run_state;
REINDEX TABLE CONCURRENTLY task_run_state;

Important note for RDS costs - Even if you shrink table files, RDS allocated storage generally cannot be decreased in-place. You’ll reclaim internal free space (future growth uses it), but your storage bill doesn’t drop unless you snapshot/restore into a smaller volume. AWS docs: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html Bottom line - Your deletes + VACUUM were correct for performance. - To see DBeaver’s size drop, run VACUUM FULL (with a lock/maintenance window) or use pg_repack. - If the goal is to lower AWS storage cost, plan a snapshot/restore to a smaller volume after compaction, since RDS doesn’t shrink allocated storage in place. If you can schedule a window, I’d: - Stop Prefect API/services and workers - VACUUM FULL the two states tables (and optionally REINDEX others if needed) - Start services back up I’m happy to help plan the exact maintenance steps for your deployment (K8s/Docker/etc.).

Well @Marvin question 2: we're trying to remove deployments in a job-clean-up (we create many deployments - for every branch, so we can try out runs even before merge, so we get lots of spammy deployments). the job is written as a gitlab-ci job, that interacts with prefect through Python API. the api works well locally, but for some weird way - the gitlab-ci run only gets two first deployments and that's it. so i guess it's not permissions / API mis-use / time-performance. Have you heard of such a weird behavior? anything i can do to debug this?

@Marvin you're alive?

I’m here and ready! How can I help with your Prefect workflows today?

Absolutely — and thanks for the details. I’ll address both for Prefect OSS 2.20. QUESTION 1 (Slack links show http://localhost:4200) This is controlled by Prefect’s UI base URL setting. Set

PREFECT_UI_URL

to your externally reachable domain (and include any path prefix if you’re serving Prefect under a subpath). Example values: -

<https://prefect.mycompany.com>

-

<https://prefect.mycompany.com/prefect>

(if you reverse proxy under /prefect) Make sure this env var is set in the Prefect API/Server pod/container (the component that emits notifications), not just your local machine. Once set, new notifications will include correct, clickable URLs. It’s not a K8s limitation — it’s a Prefect config defaulting to localhost when unset. Helpful docs: - Prefect 2 settings guide - Self-hosting Prefect 2 Quick examples: - Kubernetes (Deployment env):

- name: PREFECT_UI_URL
  value: "<https://prefect.mycompany.com>"

- Docker Compose:

environment:
  - PREFECT_UI_URL=<https://prefect.mycompany.com>

QUESTION 2 (CI job only “gets two” deployments) I haven’t seen a deterministic “2 only” bug, but a few common culprits explain this behavior: - Pagination not handled: Prefect’s deployments list is paginated. If your code doesn’t paginate explicitly, you’ll only get the first page (the default

limit

can vary by method/environment). - Different API target in CI:

PREFECT_API_URL

might be pointing to a different server/workspace in CI than locally. - Filtering differences: Filters (name/prefix/tags) can narrow to just a couple items without being obvious. - Timeout/partial iteration issues. To make the cleanup robust, explicitly paginate and/or request a large page size:

import anyio
from prefect.client.orchestration import PrefectClient

async def list_all_deployments():
    async with PrefectClient() as client:
        all_deps = []
        page_size = 200
        offset = 0
        while True:
            deps = await client.read_deployments(limit=page_size, offset=offset)
            if not deps:
                break
            all_deps.extend(deps)
            offset += len(deps)
        return all_deps

async def delete_branch_deployments(prefix: str):
    deps = await list_all_deployments()
    async with PrefectClient() as client:
        for d in deps:
            if d.name.startswith(prefix):
                await client.delete_deployment(d.id)

anyio.run(delete_branch_deployments, "your-branch-prefix-")

Debug tips: - Print the count you receive in CI vs local to confirm pagination differences. - Ensure

PREFECT_API_URL

in CI points to the same server you expect. - Turn on debug logs in CI to see the requests:

export PREFECT_LOGGING_LEVEL=DEBUG
export PREFECT_LOGGING_EXTRA_LOGGERS=httpx

- Hit the REST endpoint directly to verify pagination: - POST to

/api/deployments/filter

with

{"limit": 200, "offset": 0}

and compare total rows returned. - API reference: Prefect 2 REST API If you can share: - The snippet you use to list/delete deployments - The expected number of deployments vs what you see in CI - The value of

PREFECT_API_URL

in CI …I can help pinpoint whether it’s pagination, filtering, or environment targeting.