<@ULVA73B9P> I have a local setup with docker runn...
# ask-marvin
a
@Marvin I have a local setup with docker running prefect server and prefect database. when running: prefect server database reset -y I am receiving this error: sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: _alembic_tmp_work_queue.name [SQL: INSERT INTO _alembic_tmp_work_queue (id, created, updated, name, filter, description, is_paused, concurrency_limit, last_polled, priority, work_pool_id) SELECT work_queue.id, work_queue.created, work_queue.updated, work_queue.name, work_queue.filter, work_queue.description, work_queue.is_paused, work_queue.concurrency_limit, work_queue.last_polled, work_queue.priority, work_queue.work_pool_id FROM work_queue] (Background on this error at: https://sqlalche.me/e/20/gkpj)
m
thought for 492.7 seconds
Thanks for the details — sorry you’re hitting this. The error you’re seeing is coming from SQLite during an Alembic migration step where a temporary table
_alembic_tmp_work_queue
is created with a UNIQUE constraint on
name
. If your existing data contains duplicate work queue names (often across different work pools), the copy into that temp table fails with the UNIQUE constraint error. Why this happens - On SQLite, Alembic can’t just alter constraints; it recreates tables and copies data. If there are multiple rows with the same
work_queue.name
, the copy into the temp table fails on the unique index. - This typically shows up when resetting or upgrading a local SQLite-backed server that has multiple work queues with the same name. A couple of paths forward — pick based on whether you need to preserve data: Option A — You can wipe the DB (fastest) If you’re okay losing all server data (flows, runs, queues, etc.), delete the SQLite DB file or the Docker volume, then restart the server so it creates a clean schema. 1) Stop the Prefect server (and anything else connecting to the DB). 2) Identify the database path or connection URL: - Run:
Copy code
prefect config view --show-sources --show-defaults
Look for
PREFECT_SERVER_DATABASE_CONNECTION_URL
(alias
PREFECT_API_DATABASE_CONNECTION_URL
). If it starts with
sqlite+aiosqlite:///
, the file path follows that prefix. 3) Delete the file (or the Docker volume if you mounted one). Examples: - Direct file removal:
Copy code
rm /path/to/your/prefect.sqlite
- If you used Docker Compose and a named volume for Prefect’s home (common), remove the volume:
Copy code
docker compose down
     docker volume ls | grep prefect
     docker volume rm <your_prefect_volume_name>
4) Start the server again (or run a reset):
Copy code
prefect server database reset -y
or just start the server and it will initialize the schema. Docs: - Prefect Server CLI - Database maintenance - Settings reference Option B — You need to keep your data If you want to preserve what’s in SQLite, you’ll need to remove or rename any duplicate work queue names before running the migration/reset again. 1) Find the SQLite file path as above, then open it:
Copy code
sqlite3 /path/to/your/prefect.sqlite
2) Check for duplicates by name:
Copy code
SELECT name, COUNT(*) as cnt
   FROM work_queue
   GROUP BY name
   HAVING cnt > 1;
Optionally, check how they distribute across pools:
Copy code
SELECT name, work_pool_id, COUNT(*) as cnt
   FROM work_queue
   GROUP BY name, work_pool_id
   HAVING cnt > 1;
3) For any duplicated names, either delete or rename rows until all
name
values are unique: - Example rename approach (be careful; adjust to your data):
Copy code
-- See the rows first
     SELECT id, name FROM work_queue WHERE name = 'duplicate_name';

     -- Rename specific ones
     UPDATE work_queue
     SET name = name || '_' || id
     WHERE id IN (<ids to rename>);
4) Exit sqlite3, then try:
Copy code
prefect server database reset -y
If you prefer to only run migrations:
Copy code
prefect server database upgrade
Option C — Use Postgres for the server If you’re running a longer-lived self-hosted server, we recommend Postgres instead of SQLite. Set
PREFECT_SERVER_DATABASE_CONNECTION_URL
to a Postgres DSN, then run:
Copy code
prefect server database reset -y
to initialize. This avoids SQLite’s ALTER TABLE limitations and is more robust overall. A few checks to prevent this in the future - Make sure you’re running the CLI in the same environment/profile the server uses (so you’re resetting the intended DB). You can verify with:
Copy code
prefect config view --show-sources
- Stop the server before running resets/upgrades to avoid concurrent access to the DB file. Questions to help me tailor the next steps - Which Prefect version are you on? (Assuming Prefect 3.x here.) - Are you okay wiping the DB, or do you need to preserve data? - Are you running the server in a single Prefect container that uses SQLite by default, or do you have a Docker Compose stack with a mounted volume? - If you run
prefect config view --show-sources
, what is the value for
PREFECT_SERVER_DATABASE_CONNECTION_URL
? Once I know those, I can give you exact commands (including any Docker volume names/paths) to fix this quickly.
a
@Marvin - Which Prefect version are you on? (Assuming Prefect 3.x here.) -> 3.2.12 - Are you okay wiping the DB, or do you need to preserve data? -> yes - Are you running the server in a single Prefect container that uses SQLite by default, or do you have a Docker Compose stack with a mounted volume? -> single postgres container, mounted volume - If you run prefect config view --show-sources, what is the value for PREFECT_SERVER_DATABASE_CONNECTION_URL? -? -> 🚀 you are connected to: http://prefect-server:4200 PREFECT_PROFILE='ephemeral' PREFECT_API_URL='http://prefect-server:4200/api' (from .env file) PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='true' (from profile)
m
thought for 66.7 seconds
Thanks — that context helps a lot. Short version: your
prefect server database reset -y
is running against SQLite (hence the sqlite3 IntegrityError), but your actual server DB is Postgres. The server CLI operates directly on the database connection configured in the CLI’s environment, not via the API URL. Because your CLI environment isn’t pointing at Postgres, it’s trying to reset a default SQLite DB instead and hits the Alembic/SQLite uniqueness issue. Since you’re on Prefect 3.2.12, using Postgres, and you’re OK wiping data, here are two clean ways to reset: Option 1 — Drop the Postgres data volume (clean slate) - Stop Prefect server and Postgres containers - Remove the Postgres volume, then start fresh Example:
Copy code
docker stop <prefect_server_container> <postgres_container>

# Remove the Postgres data volume (replace with your actual volume name)
docker volume ls | grep postgres
docker volume rm <your_postgres_volume_name>

# Restart Postgres and Prefect server
docker start <postgres_container>
docker start <prefect_server_container>
When the Prefect server starts against an empty Postgres DB, it will initialize the schema. Option 2 — Run a targeted DB reset against Postgres - Stop the Prefect server to avoid active connections - Run the reset pointing explicitly at your Postgres DSN Using a one-off Prefect CLI container (recommended to ensure correct deps):
Copy code
docker stop <prefect_server_container>

docker run --rm \
  --network <your_docker_network> \
  -e PREFECT_SERVER_DATABASE_CONNECTION_URL="postgresql+asyncpg://<user>:<pass>@<postgres_host>:5432/<db_name>" \
  prefecthq/prefect:3.2.12 \
  prefect server database reset -y
Or from your host (only if your local env has asyncpg installed and can reach Postgres):
Copy code
export PREFECT_SERVER_DATABASE_CONNECTION_URL="postgresql+asyncpg://<user>:<pass>@<postgres_host>:5432/<db_name>"
prefect server database reset -y
Important notes - The API URL you shared (
PREFECT_API_URL='<http://prefect-server:4200/api>'
) only configures the HTTP client; it does not tell the
server database
CLI where your DB is. For DB operations, the CLI uses
PREFECT_SERVER_DATABASE_CONNECTION_URL
(alias:
PREFECT_API_DATABASE_CONNECTION_URL
). - To verify what the CLI will target, run this in the exact environment where you run `prefect server database reset`:
Copy code
prefect config view --show-sources --show-defaults | grep DATABASE_CONNECTION_URL
If it’s empty or shows a
sqlite+aiosqlite://
URL, that explains the SQLite error. Docs for reference: - Server CLI - Database maintenance - Settings reference (see PREFECT_SERVER_DATABASE_CONNECTION_URL) If you want, share your Postgres container name, network, and DSN (redact secrets), and I’ll provide an exact
docker run
command for Option 2.
a
@Marvin how can i wipe the prefect data in a postgres rds environment (without dropping the db )?
m
thought for 239.4 seconds
Great question. You can wipe all Prefect data in Postgres RDS without dropping the database by using either: 1) Prefect’s admin REST endpoints (recommended if you can reach the API), or 2) The Prefect CLI targeting your RDS connection URL (drops and recreates tables only). Both approaches remove all Prefect tables/data but do not drop the RDS database itself. Option A — Use the admin REST endpoints - Clear all data but keep the schema (fastest):
Copy code
curl -X POST \
    "<http(s)://<your-prefect-server>/api/admin/clear-database>" \
    -H "Content-Type: application/json" \
    -d '{"confirm": true}'
- Fully reset the schema (drop and recreate all tables):
Copy code
# Drop all Prefect tables
  curl -X POST \
    "<http(s)://<your-prefect-server>/api/admin/drop-database>" \
    -H "Content-Type: application/json" \
    -d '{"confirm": true}'

  # Recreate the schema
  curl -X POST \
    "<http(s)://<your-prefect-server>/api/admin/create-database>" \
    -H "Content-Type: application/json" \
    -d '{"confirm": true}'
Docs: - /admin/clear-database - /admin/drop-database - /admin/create-database Notes: - These are POST requests and require
{"confirm": true}
in the JSON body. - If you’ve put the API behind auth, include your auth header. - Stop your workers/agents before running this to avoid new data being written mid-operation. Option B — Use the CLI against your RDS URL This drops and recreates all Prefect tables (not the DB itself). - Stop the Prefect server (to release connections). - Run the reset with your Postgres DSN:
Copy code
# From a Prefect container (ensures asyncpg is available)
  docker run --rm \
    --network <your_network_if_needed> \
    -e PREFECT_SERVER_DATABASE_CONNECTION_URL="postgresql+asyncpg://<user>:<pass>@<rds-host>:5432/<db_name>" \
    prefecthq/prefect:3.2.12 \
    prefect server database reset -y
Docs: - Server CLI - Database maintenance What gets deleted - Flows, deployments, flow runs, task runs, logs, artifacts, blocks, UI state — everything managed by Prefect in that DB. Recommended procedure 1) Take an RDS snapshot (just in case). 2) Stop Prefect server and any workers/agents. 3) Run either the admin endpoints or the CLI reset. 4) Start Prefect server and workers again. If you share your Prefect API URL and whether your server requires auth, I can give you exact curl commands tailored to your setup.
a
@Marvin will option a remove all structs existing, e.g. deployments, automations, work pools work queues ?
m
thought for 23.5 seconds
Short answer: yes. Using the admin REST endpoints: - POST /api/admin/clear-database with {"confirm": true} - Empties all Prefect tables without dropping them. This removes every user-visible object stored in the DB: - Flows, deployments, flow runs, task runs - Automations - Work pools and work queues (and their concurrency limits) - Blocks and block documents (including stored secrets) - Artifacts, logs, events, notifications, etc. - After this, the schema remains but the database is effectively empty. - POST /api/admin/drop-database then /api/admin/create-database (each with {"confirm": true}) - Drops and recreates the schema. Functionally, this also removes everything listed above. What is not affected - Anything outside the DB (e.g., files in S3/GCS referenced by blocks, Docker images, your code repos) remains intact. - Client-side profiles and environment variables are not in the DB. Caveats - Stop your server and workers/agents before running these to avoid concurrent writes. - You’ll need to recreate work pools/queues, blocks, and redeploy flows after wiping. Links: - /admin/clear-database - /admin/drop-database - /admin/create-database