<@ULVA73B9P> our prefect api (the one we start usi...
# marvin-ai
m
@Marvin our prefect api (the one we start using the
prefect server start
command) is currently in an endless restart loop and it look like that's been happening since we merged an update that bumped the python prefect dependency from version
3.6.28
to version
3.7.1
earlier today. The logs don't show any errors or warnings, but shortly after the startup message the log line
Server stopped!
keeps appearing. Do you have any idea what might be causing this?
m
thought for 189.3 seconds
A few likely causes, yes. Important detail: the
Server stopped!
line is not itself the error. In Prefect, that message is printed in a
finally
block when the server process exits for any reason. So what you’re seeing is: the API starts, then
uvicorn
exits, then Prefect logs
Server stopped!
, and something is starting it again. I checked the Prefect 3.7.1 release notes and there’s no obvious server CLI change, but there were UI/server changes in 3.7.x that could affect startup. I also didn’t find an existing GitHub issue/discussion matching this exact symptom. Most plausible causes: 1. Startup work is failing silently On startup, Prefect Server runs things like: - DB migrations - block type registration - service startup - UI app setup If one of those exits early and stderr isn’t making it into your logs, you can get exactly this pattern. 2. Database migration / SQLite state issue If you’re using the default local DB, an upgrade can expose: - migration problems - locked SQLite DB - schema mismatch after the version bump This is probably the first thing I’d test. 3. UI-related startup regression 3.7.x included UI-serving changes. If the API works but the combined server/UI process is what’s exiting, running without the UI can help isolate it. 4. An external restart policy is masking the real failure If this is inside Docker, systemd, Kubernetes, etc., Prefect may be exiting once and your supervisor is restarting it endlessly. What I’d try next: Run it in the foreground and capture everything
Copy code
bash
prefect server start 2>&1 | tee prefect-server.log
Then look for anything just before
Server stopped!
. Try disabling non-essential startup pieces to isolate the failure
Copy code
bash
prefect server start --no-ui --no-services 2>&1 | tee prefect-server-minimal.log
If that stays up, the issue is likely in UI or a background service rather than the core API. Check the database migration path explicitly
Copy code
bash
prefect server database upgrade --dry-run
If that shows anything suspicious, then try:
Copy code
bash
prefect server database upgrade
If using SQLite, check for a locked/stale DB A common local-server culprit is the SQLite file being locked or left in a bad state by another process. Check whether something else is restarting it - Docker
restart: always
- Kubernetes liveness/readiness probes - systemd/supervisor - a wrapper script around
prefect server start
A couple of targeted tests: Test 1: no UI
Copy code
bash
prefect server start --no-ui
Test 2: no services
Copy code
bash
prefect server start --no-services
Test 3: both off
Copy code
bash
prefect server start --no-ui --no-services
Interpretation: - If only
--no-ui
fixes it, suspect a 3.7.x UI-serving issue - If only
--no-services
fixes it, suspect scheduler/related startup services - If neither fixes it, suspect DB migration/init or environment/config If this is non-production and you can afford a reset, you could also test with a fresh DB to confirm whether it’s state-related. Relevant references: - Prefect 3.7.1 release notes: Prefect 3.7.1 - Full changelog diff: 3.7.0...3.7.1 If you want, send: - how you launch it (plain shell, Docker, K8s, systemd, etc.) - the few log lines before
Server stopped!
- whether
prefect server start --no-ui --no-services
stays up and I can help narrow it down quickly.