<@ULVA73B9P> What is best practice for upgrading P...
# ask-marvin
g
@Marvin What is best practice for upgrading Postgres database gracefully?
m
thought for 108.8 seconds
For self-hosted Prefect 3.x, best practice is: Treat the Postgres upgrade as infrastructure maintenance, not a Prefect migration first. Recommended approach 1. Back up the database first 2. Stop Prefect writes gracefully: - stop workers - stop API/background services 3. Upgrade or migrate Postgres 4. Point Prefect at the upgraded Postgres instance 5. Run/verify Prefect DB migrations 6. Bring Prefect back up and validate Practical guidance 1. Take a tested backup At minimum, create a dump before touching the database. If this is production, make sure you can restore it in staging. 2. Minimize downtime by upgrading Postgres separately If you want the smoothest upgrade, the usual production pattern is: - provision a new Postgres instance on the target version - replicate or restore data into it - cut Prefect over during a maintenance window That is generally safer than doing an in-place upgrade on the only database host. 3. Stop Prefect before cutover To avoid partial writes or schema drift: - stop workers first - then stop Prefect server / API processes This keeps the database quiet before the switch. 4. Run Prefect migrations explicitly If you are also upgrading Prefect itself, run database migrations in a controlled way rather than relying on every server instance to do it on startup. You can disable auto-migration with:
Copy code
PREFECT_SERVER_DATABASE_MIGRATE_ON_START=false
Then run the migration once:
Copy code
prefect server database upgrade -y
5. For multi-server deployments If you run multiple Prefect API instances, do not let each instance race to migrate the DB. Use manual migration as above, then start the instances. 6. Watch for large tables If you have a lot of
events
,
flow_runs
, or
task_runs
, migration/index creation may take longer than expected. Plan a maintenance window and consider increasing migration timeout if needed. 7. Validate after upgrade After cutover: - confirm Prefect can connect - verify the UI/API loads - run a test flow - confirm workers pick up work - monitor DB connections and logs Important version note Prefect 3.x self-hosted deployments should use PostgreSQL 14.9+, with PostgreSQL 15+ recommended. If you want the shortest answer possible: The safest graceful upgrade is: - backup - bring up a new Postgres version - sync/restore data - stop Prefect services - switch the connection URL - run
prefect server database upgrade -y
- start Prefect again Docs - Database maintenance - Scaling self-hosted Prefect If you want, I can give you a step-by-step runbook for either: - Docker Compose self-hosted Prefect, or - Kubernetes / Helm-based Prefect