A few things stand out here: Your unit file is fine for basic restarts, but the symptom “first API call times out, second works” usually points to one of these: 1. SQLite backing the server (most common for self-hosted local setups) 2. short server/proxy keepalive timeouts 3. startup/background services contending with API requests 4. resource/network issues on the Ubuntu host For Prefect 3.x, the biggest recommendation is: Do not rely on the default SQLite setup for long-running/stable server usage. Use PostgreSQL for the Prefect API database. Relevant CLI help:

prefect server start --help

You can use:

prefect server start --host 0.0.0.0 --keep-alive-timeout 65

### Why this happens When Prefect Server is using SQLite, it can show intermittent latency/locking behavior under concurrent access. That often looks like: - first request times out - retry succeeds - UI/API sometimes feels “sleepy” or temporarily disconnected ### What I recommend #### 1) Move to PostgreSQL Set:

PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://username:password@localhost:5432/prefect>

If you stay on SQLite, long-running reliability will be worse. #### 2) Improve your systemd service Your current service works, but I’d make it more explicit:

[Unit]
Description=Prefect Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/home/shubham
Environment="PREFECT_SERVER_API_HOST=0.0.0.0"
Environment="PREFECT_SERVER_API_PORT=4200"
Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=65"
# Recommended: use PostgreSQL instead of SQLite
# Environment="PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://username:password@localhost:5432/prefect>"

ExecStart=/home/shubham/prefect-env/bin/prefect server start --host 0.0.0.0 --keep-alive-timeout 65
Restart=always
RestartSec=5
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart prefect-server
sudo systemctl status prefect-server

#### 3) Check logs Watch logs while reproducing the timeout:

journalctl -u prefect-server -f

Look for: -

database is locked

- connection reset / timeout messages - repeated restarts - migration/startup delays #### 4) If you use NGINX / load balancer, raise timeouts If there’s a reverse proxy in front of Prefect, make sure its read/connect timeout is not too low. A proxy timeout mismatch can produce exactly this “first fails, second works” behavior. #### 5) Consider separating API from services If this is a heavier setup, run the API only with:

prefect server start --host 0.0.0.0 --no-services

That can reduce contention, but database choice still matters most. ### Best-practice summary For a stable long-running Ubuntu deployment: - Use PostgreSQL - run Prefect with systemd - set

--keep-alive-timeout 65

- inspect

journalctl

logs - avoid SQLite for production-ish usage ### Helpful docs / references - Prefect docs - API/reference docs If you want, I can help you with either: 1. a production-ready systemd service file, or 2. a PostgreSQL-backed Prefect Server setup on Ubuntu step by step.

That strongly suggests the server is becoming temporarily unavailable, not just the browser UI. Most likely causes, in order: 1. SQLite lock/contention 2. very short default keepalive timeout 3. server under load / blocking requests In Prefect 3.x, the default API keepalive timeout is only 5 seconds, which is low for a self-hosted server. ## Try this first Update your systemd unit to increase timeouts:

[Unit]
Description=Prefect Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/home/shubham
Environment="PREFECT_SERVER_API_HOST=0.0.0.0"
Environment="PREFECT_SERVER_API_PORT=4200"
Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=300"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_INTERVAL=30"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_TIMEOUT=30"

ExecStart=/home/shubham/prefect-env/bin/prefect server start --host 0.0.0.0 --keep-alive-timeout 300
Restart=always
RestartSec=5
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Then run:

sudo systemctl daemon-reload
sudo systemctl restart prefect-server
journalctl -u prefect-server -f

## Very important: are you using SQLite? If you did not set

PREFECT_API_DATABASE_CONNECTION_URL

, then you're probably using the default SQLite DB. That is the most likely reason for: - UI disconnecting after a few minutes - API timing out - restart fixing it temporarily ### Best fix Switch Prefect Server to PostgreSQL. Example:

Environment="PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://username:password@localhost:5432/prefect>"

## Check health when UI disconnects When it happens, test these from the Ubuntu server:

curl <http://127.0.0.1:4200/api/health>
curl <http://127.0.0.1:4200/api/ready>

If

/api/health

works but

/api/ready

fails or hangs, that points to the database. ## Check logs for DB lock symptoms Run:

journalctl -u prefect-server -n 200 --no-pager

Look for: -

database is locked

-

SQLITE_BUSY

-

503 Service Unavailable

## Short answer If restart fixes it only for a few minutes, I would strongly suspect SQLite. Increasing keepalive may help, but moving to PostgreSQL is the real fix. If you want, I can give you: - a PostgreSQL install + Prefect config guide for Ubuntu, or - a final production-ready systemd service file for your machine.

That makes it even more likely this is server-side blocking, not just a UI websocket issue. If

GET /api/health

times out on the first attempt, then the Prefect API process is intermittently not responding fast enough at all. The usual causes are: - SQLite/database contention - CPU or memory pressure on the host - the single

prefect server start

process getting blocked briefly ## Most likely root cause If you're using the default DB, it's probably SQLite, and that is not a good long-running backend for a busy/self-hosted Prefect server. A restart helps because it clears the temporary blocked state, but after a few minutes the problem returns. ## Please check this first On the Ubuntu machine, run:

/home/shubham/prefect-env/bin/prefect config view

Look for:

PREFECT_API_DATABASE_CONNECTION_URL

If it shows

sqlite...

or is unset, that's the issue I'd fix first. ## Immediate actions ### 1) Increase server timeout settings Even if SQLite is the issue, do this too:

[Unit]
Description=Prefect Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/home/shubham
Environment="PREFECT_SERVER_API_HOST=0.0.0.0"
Environment="PREFECT_SERVER_API_PORT=4200"
Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=300"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_INTERVAL=30"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_TIMEOUT=30"

ExecStart=/home/shubham/prefect-env/bin/prefect server start --host 0.0.0.0 --keep-alive-timeout 300
Restart=always
RestartSec=5
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

### 2) Watch logs live while reproducing

journalctl -u prefect-server -f

### 3) Check machine resources when it hangs

top
free -h
df -h

If RAM is very low or CPU is pegged, that can also explain health timeouts. ## Best fix: use PostgreSQL Set:

PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://username:password@localhost:5432/prefect>

Then restart the service. ## Also useful When it is failing, test locally from the same machine:

curl -v <http://127.0.0.1:4200/api/health>

If localhost itself times out, then the issue is definitely the Prefect process / DB / host resources, not network. ## My recommendation At this point I would not spend much more time tuning SQLite. I would: 1. confirm the DB URL 2. move to PostgreSQL 3. keep the longer keepalive settings If you want, send me the output of:

/home/shubham/prefect-env/bin/prefect config view
journalctl -u prefect-server -n 100 --no-pager

and I can tell you exactly what to change.

If it happens even with no workloads and at a fixed interval every time, that points away from SQLite load and more toward a network / timeout / proxy / browser polling pattern. Most likely possibilities now are: 1. something in front of Prefect is closing idle connections on a fixed timer - NGINX - cloud firewall / load balancer - corporate proxy - SSH tunnel 2. server keepalive timeout mismatch 3. less likely: a periodic service/network interruption on the host ## Important distinction When you say

/api/health

times out on first attempt: - does it time out from the browser machine only? - or does it also time out when run locally on the Ubuntu server with

curl 127.0.0.1:4200/api/health

? This is the key test. ## Please test these two commands ### From the Ubuntu server itself

date
curl -v <http://127.0.0.1:4200/api/health>

### From your local machine/browser side

date
curl -v http://<ubuntu-server-ip>:4200/api/health

Run them when the disconnect is happening. ## How to interpret ### Case 1: local curl on Ubuntu works, remote curl fails/times out Then Prefect itself is fine, and the issue is: - firewall - proxy - load balancer - NAT/session timeout - browser/network path ### Case 2: local curl on Ubuntu also times out Then Prefect process itself is stalling, and we should look at: - DB backend - process state - resource starvation - server startup mode ## Because the interval is fixed A fixed interval strongly suggests timeout configuration. Prefect server has a very short default keepalive timeout. Set it much higher:

ExecStart=/home/shubham/prefect-env/bin/prefect server start --host 0.0.0.0 --keep-alive-timeout 300
Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=300"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_INTERVAL=30"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_TIMEOUT=30"

## One more strong possibility If you're opening the UI through something like: -

ssh -L

- VS Code port forwarding - cloud VM browser preview - a platform tunnel those often disconnect at a very regular interval. ## Recommended next step Please tell me: 1. How are you accessing the UI? - direct

<http://server-ip:4200>

- nginx - ssh tunnel - cloud VM forwarded port - docker port mapping 2. When disconnected, what happens for:

curl <http://127.0.0.1:4200/api/health>

run on the Ubuntu box itself? That will let me narrow it down quickly.

That explains it — Prefect is fine locally; the problem is the Cloudflare tunnel/path, not the Prefect server process. Because: -

curl <http://127.0.0.1:4200/api/health>

on Ubuntu always works - remote access fails intermittently - failure happens on a regular interval - SSH stays up This points to Cloudflare tunnel/proxy timeout or websocket/connection handling, which affects the Prefect UI a lot because the UI is chatty and uses long-lived connections. ## What’s happening Prefect UI talks to the API continuously. Through Cloudflare Tunnel, you can see: - idle/keepalive connections being closed - websocket upgrade issues - proxy read timeout behavior - first request after idle timing out, second succeeding after reconnect That matches your symptoms exactly. ## Good news You likely do not have a broken Prefect install. ## What to do ### 1) Increase Prefect keepalive anyway Keep this in your service:

Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=300"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_INTERVAL=30"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_TIMEOUT=30"

ExecStart=/home/shubham/prefect-env/bin/prefect server start --host 0.0.0.0 --keep-alive-timeout 300

### 2) Check Cloudflare Tunnel config Make sure your tunnel ingress is a plain HTTP service to the local Prefect port, for example:

tunnel: <tunnel-id>
credentials-file: /root/.cloudflared/<tunnel-id>.json

ingress:
  - hostname: <http://prefect.yourdomain.com|prefect.yourdomain.com>
    service: <http://localhost:4200>
  - service: http_status:404

If you're doing anything more complex in front of it, simplify it. ### 3) Disable/avoid extra proxy layers If you have: - Cloudflare Tunnel -> NGINX -> Prefect try temporarily testing: - Cloudflare Tunnel -> Prefect directly ### 4) Test without Cloudflare From your own machine, test direct connectivity if possible: - VPN - private IP - SSH local port forward For example:

ssh -L 4200:localhost:4200 user@your-server

Then open:

<http://localhost:4200>

If this is stable, that confirms Cloudflare tunnel is the cause. ### 5) Consider using SSH tunnel or VPN for admin UI access For a private self-hosted Prefect UI, SSH tunnel or Tailscale/WireGuard is often more reliable than Cloudflare Tunnel for websocket-heavy internal tools. ## Recommended systemd unit

[Unit]
Description=Prefect Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/home/shubham
Environment="PREFECT_SERVER_API_HOST=0.0.0.0"
Environment="PREFECT_SERVER_API_PORT=4200"
Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=300"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_INTERVAL=30"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_TIMEOUT=30"

ExecStart=/home/shubham/prefect-env/bin/prefect server start --host 0.0.0.0 --keep-alive-timeout 300
Restart=always
RestartSec=5
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

## My conclusion Since localhost is always healthy, don’t focus on restarting Prefect anymore. Focus on the Cloudflare tunnel behavior. ## Next best test Try this exact comparison: ### Stable path

ssh -L 4200:localhost:4200 user@server

Open

<http://localhost:4200>

### Current path Open through Cloudflare tunnel hostname If SSH-forwarded UI is stable but Cloudflare hostname disconnects, you've isolated the issue. If you want, I can help you with: - a Cloudflare Tunnel config review for Prefect UI - or a more reliable private access setup using SSH tunnel / Tailscale / NGINX.

Yes — add the timeout environment/config. It’s the first thing I’d change now. Since: - localhost on the server works - remote/local-network access shows disconnects after a fixed interval - you have not set the timeout yet the default keepalive timeout is very likely too low for your access pattern. ## What

300

means

PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=300

means: - the server keeps an idle HTTP connection open for 300 seconds - instead of closing it after the default 5 seconds So

300

= 5 minutes. This does not make each request take 300 seconds. It just means the server won’t close an idle persistent connection too aggressively. That matters because browsers and UI apps reuse connections. If the server closes them too quickly, you can get: - disconnect banners - first request failing after idle - second request succeeding after reconnect ## What to set Add these:

Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=300"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_INTERVAL=30"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_TIMEOUT=30"

and also pass the CLI flag:

ExecStart=/home/shubham/prefect-env/bin/prefect server start --host 0.0.0.0 --keep-alive-timeout 300

## Updated service file

[Unit]
Description=Prefect Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/home/shubham
Environment="PREFECT_SERVER_API_HOST=0.0.0.0"
Environment="PREFECT_SERVER_API_PORT=4200"
Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=300"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_INTERVAL=30"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_TIMEOUT=30"

ExecStart=/home/shubham/prefect-env/bin/prefect server start --host 0.0.0.0 --keep-alive-timeout 300
Restart=always
RestartSec=5
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Then:

sudo systemctl daemon-reload
sudo systemctl restart prefect-server
sudo systemctl status prefect-server

## Why set both env var and CLI flag? You technically only need one, but setting both is fine and makes the intended value obvious. ## If this fixes it Then the root cause was connection timeout/keepalive behavior. ## If this does not fix it Then next things to inspect are: 1. browser devtools network errors 2. whether something on the LAN is dropping idle connections 3. SQLite/backend issues 4. Prefect version-specific behavior ## One more useful check See which Prefect version you’re on:

/home/shubham/prefect-env/bin/prefect version

If you send me: - your Prefect version - updated service file - whether

300

changed the behavior I can help narrow it down further.

That pattern is very helpful. If every first request after ~30–40 seconds of inactivity is slow/fails, and the next request works, that usually means: the existing connection is being dropped somewhere, and the client only recovers on retry/reconnect. Since you also see it on the API, this still sounds like connection keepalive / stale connection reuse, not a crashed Prefect process. ## Important note about

300

Setting

PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=300

should help only if the server is actually starting with that value. So first, confirm the running config. ## Please check the actual config Run on the server:

/home/shubham/prefect-env/bin/prefect config view | grep -i KEEPALIVE

and also:

systemctl cat prefect-server

I want to confirm the service is really using:

--keep-alive-timeout 300

## What I suspect now A 30–40 second interval is very suggestive of: - browser or client trying to reuse a stale HTTP connection - websocket/polling connection expiring - some network component timing out idle sessions - or Prefect/Uvicorn keepalive mismatch if the setting hasn’t actually been applied ## Best next test After updating the service, compare these: ### 1) Immediate request

curl -v http://<server-ip>:4200/api/health

### 2) Wait 40 seconds, then run again

curl -v http://<server-ip>:4200/api/health

If the second one after idle stalls, but an immediate retry works, that strongly confirms stale connection/idle timeout behavior. ## You should also try forcing new connections From another machine:

curl -v --http1.1 -H "Connection: close" http://<server-ip>:4200/api/health

Wait 40 seconds and run it again. If

Connection: close

avoids the issue, then the problem is definitely persistent-connection reuse. ## Recommended service file Use this exactly:

[Unit]
Description=Prefect Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/home/shubham
Environment="PREFECT_SERVER_API_HOST=0.0.0.0"
Environment="PREFECT_SERVER_API_PORT=4200"
Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=300"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_INTERVAL=30"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_TIMEOUT=30"

ExecStart=/home/shubham/prefect-env/bin/prefect server start --host 0.0.0.0 --keep-alive-timeout 300
Restart=always
RestartSec=5
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Then reload:

sudo systemctl daemon-reload
sudo systemctl restart prefect-server

## One more thing If this still happens even with

Connection: close

, then we should inspect whether a firewall/NAT on the Ubuntu host or LAN is doing 30-second idle cleanup. ## My guess right now Most likely: - the timeout setting is not yet applied, or - the client/network path is reusing a dead idle connection every ~30–40 sec If you send me the outputs of:

/home/shubham/prefect-env/bin/prefect config view | grep -i -E "KEEPALIVE|WEBSOCKET"
/home/shubham/prefect-env/bin/prefect version
systemctl cat prefect-server

I can tell you if the service is configured correctly.

300

should not make things worse by itself. What it does is only for idle HTTP keep-alive connections. It does not directly control the UI websocket connection. So if after setting

300

you're now seeing disconnects in under 5 seconds, that usually means: - the setting is not the real issue, or - another layer is closing the connection, or - the UI websocket path is the one failing ## Important correction There are two different things here: 1. HTTP keepalive timeout

PREFECT_SERVER_API_KEEPALIVE_TIMEOUT

- affects idle HTTP connections - helps with “first request after idle fails, retry works” 2. WebSocket ping settings

PREFECT_SERVER_API_WEBSOCKET_PING_INTERVAL

PREFECT_SERVER_API_WEBSOCKET_PING_TIMEOUT

- affects the UI’s live connection behavior So

300

is not a “request timeout”. It just tells the server: don’t close idle HTTP connections for 300 seconds. ## Since it got worse That suggests one of these: - the service didn’t reload the way expected - the problem is actually websocket-related - there is some network/proxy issue between browser and server - possibly browser stale state/cache ## What I’d try next ### Use these values instead Don’t use huge numbers yet. Try:

Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=60"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_INTERVAL=10"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_TIMEOUT=10"

and:

ExecStart=/home/shubham/prefect-env/bin/prefect server start --host 0.0.0.0 --keep-alive-timeout 60

That means: - HTTP idle connection kept for 60 sec - websocket ping every 10 sec - websocket considered dead after 10 sec without pong ### Full service

[Unit]
Description=Prefect Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/home/shubham
Environment="PREFECT_SERVER_API_HOST=0.0.0.0"
Environment="PREFECT_SERVER_API_PORT=4200"
Environment="PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=60"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_INTERVAL=10"
Environment="PREFECT_SERVER_API_WEBSOCKET_PING_TIMEOUT=10"

ExecStart=/home/shubham/prefect-env/bin/prefect server start --host 0.0.0.0 --keep-alive-timeout 60
Restart=always
RestartSec=5
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Then:

sudo systemctl daemon-reload
sudo systemctl restart prefect-server

## Also verify the settings are actually active Run:

/home/shubham/prefect-env/bin/prefect config view | grep -i -E "KEEPALIVE|WEBSOCKET"
systemctl cat prefect-server

## Very useful test Try the API from another machine with connection reuse disabled:

curl -v --http1.1 -H "Connection: close" http://<server-ip>:4200/api/health

Then wait 30–40 seconds and run it again. ### If this works consistently Then the problem is very likely persistent connection reuse / stale connection behavior. ### If this still fails Then the problem is somewhere else in the browser/network path. ## My practical recommendation At this point I would test in this order: 1. set keepalive to

60

, not

300

2. set websocket ping interval/timeout to

10

3. test with

curl -H "Connection: close"

4. open browser devtools and check network failures If you want, send me these outputs next:

/home/shubham/prefect-env/bin/prefect config view | grep -i -E "KEEPALIVE|WEBSOCKET"
/home/shubham/prefect-env/bin/prefect version
curl -v --http1.1 -H "Connection: close" http://<server-ip>:4200/api/health

and I’ll help interpret them.