<@ULVA73B9P> whenever i try to bulk delete flows f...
# ask-marvin
n
@Marvin whenever i try to bulk delete flows from the ui, i get 503 service unavailable errors in the api and get a toast saying oops something went wrong
m
thought for 336.9 seconds
Thanks for reporting this — sorry for the rough edges. A 503 during bulk delete usually means the API request is timing out or the backend is temporarily unavailable. To help us zero in on the cause, could you share: - Are you on Prefect Cloud or self-hosted? (If self-hosted, which server version?) - Approximately how many flows are you selecting when it fails? - Does deleting a single flow from the UI work? - If you can check your browser’s dev tools Network tab, which endpoint returns 503 and what’s the response body (if any)? - Rough timestamp and workspace/org (feel free to DM if sensitive) so we can check logs. Quick workarounds in the meantime - Try smaller batches in the UI (e.g., 5–10 at a time) to see if there’s a threshold that triggers 503s. - Use the Python client to delete in chunks. In Prefect 3.x, you can do: Async (recommended)
Copy code
import asyncio
from prefect.client.orchestration import get_client
from prefect.client.schemas.filters import FlowFilter, FlowFilterName

async def delete_flows_by_prefix(prefix: str, batch_size: int = 10):
    async with get_client() as client:
        offset = 0
        total_deleted = 0
        while True:
            # like_ does a case-insensitive substring match
            flows = await client.read_flows(
                flow_filter=FlowFilter(name=FlowFilterName(like_=prefix)),
                limit=batch_size,
                offset=offset,
            )
            if not flows:
                break

            for f in flows:
                await client.delete_flow(f.id)
                total_deleted += 1

            offset += batch_size

        print(f"Deleted {total_deleted} flows")

if __name__ == "__main__":
    asyncio.run(delete_flows_by_prefix("your-prefix-"))
Sync
Copy code
from prefect.client.orchestration import get_client
from prefect.client.schemas.filters import FlowFilter, FlowFilterName

def delete_flows_by_prefix_sync(prefix: str, batch_size: int = 10):
    with get_client(sync_client=True) as client:
        offset = 0
        total_deleted = 0
        while True:
            flows = client.read_flows(
                flow_filter=FlowFilter(name=FlowFilterName(like_=prefix)),
                limit=batch_size,
                offset=offset,
            )
            if not flows:
                break

            for f in flows:
                client.delete_flow(f.id)
                total_deleted += 1

            offset += batch_size

        print(f"Deleted {total_deleted} flows")

delete_flows_by_prefix_sync("your-prefix-")
If you already have specific flow IDs you want to remove, you can iterate those directly instead of filtering:
Copy code
from uuid import UUID
from prefect.client.orchestration import get_client

ids = [
    "11111111-1111-1111-1111-111111111111",
    "22222222-2222-2222-2222-222222222222",
]

def delete_by_ids_sync(ids):
    with get_client(sync_client=True) as client:
        for _id in ids:
            client.delete_flow(UUID(_id))

delete_by_ids_sync(ids)
Next steps - If you can share the failing endpoint and approximate counts, I can try to reproduce and file a bug with our team. - If this is Cloud and it repros even for small batches (e.g., <10 flows), please DM your org/workspace and a recent timestamp — we’ll investigate backend logs right away.
n
• self hosted running locally • can fail when deleting for 5 runs • no issues deleting one run • all the delete endpoints response is ◦ {“exception_message”:“Service Unavailable”} with 503 • fails starting at 2 runs
@Marvin • self hosted running locally • can fail when deleting for 5 runs • no issues deleting one run • all the delete endpoints response is ◦ {“exception_message”:“Service Unavailable”} with 503 • fails starting at 2 runs
m
thought for 248.1 seconds
Thanks for the details — that helps a lot. It sounds like you’re bulk deleting flow runs (not flows), and on self‑hosted (likely SQLite), the API starts returning 503 once there’s more than one delete happening at the same time. That pattern is very consistent with SQLite write locks when multiple delete requests are issued concurrently (the UI typically fires one DELETE per selected item in parallel). With SQLite, concurrent writes can easily trigger transient failures; in Prefect’s API these can surface as 503 Service Unavailable. What you can do right now - Prefer the bulk delete endpoint so the server does the work in a single transaction instead of many parallel deletes: - HTTP:
Copy code
curl -X POST "$PREFECT_API_URL/flow_runs/bulk_delete" \
      -H "Content-Type: application/json" \
      -d '{"flow_runs":{"id":{"any_":["<uuid-1>","<uuid-2>"]}},"limit":50}'
- Python (async):
Copy code
import asyncio
    from prefect.client.orchestration import get_client

    async def bulk_delete_run_ids(ids):
        async with get_client() as client:
            resp = await client.request(
                "POST",
                "/flow_runs/bulk_delete",
                json={"flow_runs": {"id": {"any_": ids}}, "limit": min(50, len(ids))},
            )
            print("Deleted:", resp.json().get("deleted", []))

    asyncio.run(bulk_delete_run_ids(["<uuid-1>", "<uuid-2>"]))
- Python (filter + sequential delete if you don’t have IDs handy):
Copy code
import asyncio
    from prefect.client.orchestration import get_client
    from prefect.client.schemas.filters import FlowRunFilter, FlowFilter, FlowFilterName

    async def delete_runs_for_flow_prefix(prefix, limit=10):
        async with get_client() as client:
            runs = await client.read_flow_runs(
                flow_filter=FlowFilter(name=FlowFilterName(like_=prefix)),
                limit=limit,
            )
            for r in runs:
                await client.delete_flow_run(r.id)

    asyncio.run(delete_runs_for_flow_prefix("my-flow-prefix", limit=25))
- In the UI, try very small batches (2–5). If it still fails at 2, stick to one at a time or use the bulk endpoint/script above. - For a more durable fix in self‑hosted, switch from SQLite to Postgres. SQLite doesn’t handle concurrent writes well; Postgres will. If you’re using Docker Compose, point the API to a Postgres service and re-run migrations. A couple of quick diagnostics - Can you confirm you’re seeing this specifically with flow runs (not flows)? The UI uses different endpoints for flows vs. flow runs. - Check the API server logs at the moment of failure — if you see “database is locked” or SQLAlchemy/OperationalError, that confirms SQLite contention. - Try this experiment: use the bulk endpoint with two IDs in a single request. If that succeeds but selecting two in the UI fails, it’s almost certainly UI parallel deletes + SQLite lock contention. If you can share: - Your server version (
prefect version
), - How you’re running the server (CLI vs Docker, SQLite vs Postgres), - A short snippet from the API logs when the 503s occur, …I can reproduce and get a bug filed for the UI to prefer the bulk endpoint (or serialize deletes) when multiple runs are selected. In the meantime, using the bulk endpoint or a short script to delete sequentially should unblock you.
n
@Marvin changing sqlite to psql fixed the issue
m
thought for 78.7 seconds
Great to hear — that lines up with what we see: SQLite’s write locking can cause transient failures when the UI issues multiple parallel deletes. Postgres handles concurrent writes much better, so switching resolves the 503s. If it helps for future setup/tuning: - Set the DB URL - SQLite (default):
PREFECT_API_DATABASE_CONNECTION_URL=sqlite+aiosqlite:///.../prefect.db
- Postgres:
PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@host:5432/dbname>
- Useful Postgres pool settings (adjust for your deployment size) -
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=10
-
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
-
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=30
-
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=3600
- Optional timeouts/logging -
PREFECT_API_DATABASE_TIMEOUT=30.0
-
PREFECT_API_DATABASE_CONNECTION_TIMEOUT=5.0
-
PREFECT_API_DATABASE_ECHO=false
Docs: - Self-hosted setup - Database maintenance - Settings reference If you ever need bulk deletions again, using the flow-run bulk delete endpoint in a single request per batch will also help avoid parallel-write contention, even on Postgres:
Copy code
POST /flow_runs/bulk_delete
{"flow_runs":{"id":{"any_":["<uuid-1>","<uuid-2>"]}},"limit":50}
Glad you’re unblocked — if you want, I can also file a UX ticket for the UI to prefer the bulk endpoint when multiple items are selected, which would make this smoother for SQLite users too.