Shubham
07/07/2026, 3:12 PMMarvin
07/07/2026, 3:12 PMMarvin
07/07/2026, 3:14 PMRestart=always, so if Prefect exits/crashes, systemd will keep bringing it back, but that does not fix the underlying issue.
2. prefect server start is fine for testing/small setups, but for a long-running server on Ubuntu, the first thing to check is why it is dying or becoming unhealthy.
3. Your current service file has no logging settings and no explicit database/backend config, so it may be defaulting to SQLite, which can become unstable under concurrent usage.
Most likely causes
- Prefect process is crashing after startup
- SQLite locking / DB issues
- background services inside prefect server start are failing
- reverse proxy / keep-alive timeout mismatch if you access it through nginx or LB
First: check the actual logs
Run these right after it “disappears”:
bash
sudo systemctl status prefect-server
sudo journalctl -u prefect-server -n 200 --no-pager
sudo journalctl -u prefect-server -f
Also check whether the port is still listening:
bash
ss -lntp | grep 4200
If nothing is listening on 4200, the process likely died.
Your current service file is too minimal
I would change it to this as a safer baseline:
ini
[Unit]
Description=Prefect Server
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=root
WorkingDirectory=/home/ubuntu/prefect-server
Environment="PREFECT_SERVER_API_HOST=0.0.0.0"
Environment="PREFECT_SERVER_API_PORT=4200"
Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=60"
Environment="PREFECT_SERVER_LOGGING_LEVEL=DEBUG"
ExecStart=/home/ubuntu/prefect-server/prefect-env/bin/prefect server start --host 0.0.0.0 --port 4200 --keep-alive-timeout 60
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
Then reload and restart:
bash
sudo systemctl daemon-reload
sudo systemctl restart prefect-server
sudo systemctl status prefect-server
Important note about SQLite
If you did not configure PREFECT_API_DATABASE_CONNECTION_URL, Prefect Server may be using SQLite by default. That can work for light/local use, but it is a common cause of instability.
For a more reliable setup, use PostgreSQL:
ini
Environment="PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://USER:PASSWORD@HOST:5432/prefect>"
Better production-style approach
Instead of running everything in one process, run:
- API only:
bash
prefect server start --no-services
- background services separately:
bash
prefect server services start
That means two systemd services instead of one. This is generally more stable.
Why this may “work for a few minutes”
That pattern often means:
- startup succeeds
- some background component fails later
- DB locks accumulate
- proxy timeout disconnects clients
- process exits and systemd restarts it
If you use nginx
If Prefect is behind nginx, check proxy timeout settings too. A too-short timeout can make Prefect look dead even when the process is still up.
Useful CLI reference
prefect server start --help confirms these options exist:
- --host
- --port
- --keep-alive-timeout
- --log-level
- --no-services
Relevant docs / references
- Prefect docs: Prefect documentation
- Prefect repo: GitHub repo
If you want, send me the output of:
bash
sudo journalctl -u prefect-server -n 200 --no-pager
and I can help identify the exact failure.Shubham
07/07/2026, 3:23 PMMarvin
07/07/2026, 3:23 PMMarvin
07/07/2026, 3:23 PMShubham
07/07/2026, 3:24 PMMarvin
07/07/2026, 3:24 PMMarvin
07/07/2026, 3:24 PMephemeral
- Server type: ephemeral
- Database: sqlite
This is very likely the root cause.
What this means
An ephemeral Prefect server is meant for temporary/local use, not a stable long-running system service. Combined with SQLite, it can easily become unavailable after some activity.
For a persistent Ubuntu service, you should switch to a real persistent server configuration, especially:
- a non-ephemeral profile
- PostgreSQL instead of SQLite
Why your current setup is fragile
- ephemeral mode is not intended for production-like hosting
- SQLite is not a good long-running multi-request backend for Prefect Server
- restarting the service temporarily clears the bad state, so it appears to “work for a few minutes”
Recommended fix
Use a persistent profile and configure PostgreSQL.
Step 1: create a real Prefect home
For example:
bash
mkdir -p /home/ubuntu/.prefect
Step 2: install and run PostgreSQL
If PostgreSQL is not already installed:
bash
sudo apt update
sudo apt install postgresql postgresql-contrib
Create database/user:
bash
sudo -u postgres psql
Then inside psql:
sql
CREATE DATABASE prefect;
CREATE USER prefect WITH PASSWORD 'prefectpassword';
GRANT ALL PRIVILEGES ON DATABASE prefect TO prefect;
\q
Step 3: update your systemd service
Use something like this:
ini
[Unit]
Description=Prefect Server
After=network-online.target postgresql.service
Wants=network-online.target
[Service]
Type=simple
User=root
WorkingDirectory=/home/ubuntu/prefect-server
Environment="PREFECT_HOME=/home/ubuntu/.prefect"
Environment="PREFECT_SERVER_API_HOST=0.0.0.0"
Environment="PREFECT_SERVER_API_PORT=4200"
Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=60"
Environment="PREFECT_SERVER_LOGGING_LEVEL=DEBUG"
Environment="PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://prefect:prefectpassword@127.0.0.1:5432/prefect>"
ExecStart=/home/ubuntu/prefect-server/prefect-env/bin/prefect server start --host 0.0.0.0 --port 4200 --keep-alive-timeout 60
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
Then reload:
bash
sudo systemctl daemon-reload
sudo systemctl restart prefect-server
sudo systemctl status prefect-server
Step 4: verify it is no longer ephemeral
Run:
bash
/home/ubuntu/prefect-server/prefect-env/bin/prefect version
You want to stop seeing:
- Profile: ephemeral
- Server type: ephemeral
Step 5: check API health
bash
curl <http://127.0.0.1:4200/api/health>
If exposed externally:
bash
curl <http://YOUR_SERVER_IP:4200/api/health>
Optional but recommended
For better stability, separate the API and services:
- API:
bash
prefect server start --no-services
- background services:
bash
prefect server services start
That means 2 systemd units instead of 1.
Very important
If you are connecting to this server from clients/agents/workers, also set:
bash
PREFECT_API_URL=<http://YOUR_SERVER_IP:4200/api>
What I think is happening right now
Your systemd unit starts Prefect, but because it is using an ephemeral + SQLite setup, it is acting like a temporary local server rather than a durable service. That is why it comes back after restart, then fails again.
If you want, I can give you:
1. a complete corrected systemd file for single-service Prefect Server, or
2. a better 2-service setup with separate API + background services.Shubham
07/07/2026, 4:06 PMMarvin
07/07/2026, 4:06 PMMarvin
07/07/2026, 4:06 PMephemeral profile
- ephemeral server type
- sqlite database
That combination is okay for local testing, but not for a long-running Ubuntu service.
What we changed conceptually:
- set a persistent PREFECT_HOME
- stopped relying on the ephemeral/default profile
- configured a real database backend instead of SQLite
- improved the systemd service so logging/restarts are more controlled
- increased keep-alive timeout to reduce connection drops
Why it helped
- Ephemeral mode is temporary by design
- SQLite can become unstable under concurrent requests
- PostgreSQL is much better for a continuously running Prefect server
- better systemd settings make failures easier to detect and recover from
Ongoing maintenance: what to take care of
1. Watch the service logs
Regularly check:
bash
sudo journalctl -u prefect-server -n 100 --no-pager
For live logs:
bash
sudo journalctl -u prefect-server -f
Look for:
- database connection errors
- repeated restarts
- timeout errors
- migration failures
2. Monitor service health
Check if the API is healthy:
bash
curl <http://127.0.0.1:4200/api/health>
You can also confirm the service is active:
bash
sudo systemctl status prefect-server
3. Maintain PostgreSQL
Now that PostgreSQL is the backend, ongoing care includes:
- make regular backups
- monitor disk usage
- ensure PostgreSQL service is running
- rotate logs if needed
Basic checks:
bash
sudo systemctl status postgresql
df -h
4. Avoid going back to SQLite / ephemeral
If Prefect starts again in ephemeral mode, you may see the same issue return. Periodically confirm:
bash
/home/ubuntu/prefect-server/prefect-env/bin/prefect version
You should not see:
- Profile: ephemeral
- Server type: ephemeral
5. Be careful during upgrades
When upgrading Prefect:
- upgrade in a maintenance window
- back up PostgreSQL first
- restart the service after upgrade
- watch logs for migration issues
Example:
bash
source /home/ubuntu/prefect-server/prefect-env/bin/activate
pip install -U prefect
sudo systemctl restart prefect-server
Then verify:
bash
sudo journalctl -u prefect-server -n 100 --no-pager
6. Check systemd restart behavior
Sometimes repeated crashes can get hidden because systemd keeps restarting the service. Check restart counts with:
bash
sudo systemctl status prefect-server
If it keeps restarting, inspect logs immediately.
7. If using nginx or a reverse proxy
Keep timeout settings aligned. Otherwise Prefect may appear unavailable even when it is running.
8. Consider separating API and services later
If usage grows, a better setup is:
- one service for prefect server start --no-services
- another for prefect server services start
That makes troubleshooting easier and improves stability.
Simple maintenance checklist
- check systemctl status prefect-server
- check /api/health
- review logs
- back up PostgreSQL
- watch disk space and memory
- verify Prefect upgrades carefully
Good practice commands
bash
sudo systemctl status prefect-server
sudo journalctl -u prefect-server -n 100 --no-pager
curl <http://127.0.0.1:4200/api/health>
sudo systemctl status postgresql
df -h
If you want, I can also give you a small weekly maintenance checklist or a backup + monitoring script for your Prefect server.Shubham
07/07/2026, 4:10 PMMarvin
07/07/2026, 4:11 PMMarvin
07/07/2026, 4:12 PMbash
PREFECT_SERVER_EVENTS_RETENTION_PERIOD
Default:
- 7 days
### 2. Flow run cleanup retention
This controls cleanup of old flow runs and their orphaned logs/artifacts:
bash
PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD
Default:
- 90 days
But important: flow run cleanup is not active unless enabled.
You must enable vacuum cleanup with:
bash
PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=events,flow_runs
If you leave it at default, typically only event cleanup is enabled.
---
What gets auto-cleaned
When flow-run vacuum is enabled, Prefect can clean up:
- old flow runs
- orphaned logs
- orphaned artifacts
For logs and artifacts, there is not usually a separate retention setting — they are cleaned when their related flow runs are removed.
---
Example systemd settings
You can add these to your service file:
ini
Environment="PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=events,flow_runs"
Environment="PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD=30d"
Environment="PREFECT_SERVER_EVENTS_RETENTION_PERIOD=7d"
Environment="PREFECT_SERVER_SERVICES_DB_VACUUM_LOOP_SECONDS=3600"
Or if your version expects seconds, use explicit seconds:
ini
Environment="PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD=2592000"
Environment="PREFECT_SERVER_EVENTS_RETENTION_PERIOD=604800"
Then reload systemd:
bash
sudo systemctl daemon-reload
sudo systemctl restart prefect-server
---
Recommended practical setup
A common setup is:
ini
Environment="PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED=events,flow_runs"
Environment="PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD=30d"
Environment="PREFECT_SERVER_EVENTS_RETENTION_PERIOD=7d"
That means:
- keep flow/task run history for 30 days
- keep events for 7 days
- clean related logs/artifacts as old runs are deleted
---
How to verify current settings
Run:
bash
prefect config view | grep -i "retention\|vacuum"
and check logs for vacuum activity:
bash
sudo journalctl -u prefect-server -f
---
Important caveat
Prefect does not automatically clean everything. Things like:
- deployments
- work pools
- blocks
- variables
- some metadata
are not generally pruned by retention settings.
---
Relevant docs
- Database maintenance: Database maintenance
- Self-hosted server docs: Self-hosted Prefect
- Settings reference: Settings reference
If you want, I can help you create the exact retention lines to add into your current systemd service for:
- 7 days
- 30 days
- 90 days
depending on how much history you want to keep.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by