< Marvin> im running prefect in distributed way on same vm w Prefect Community #ask-marvin

<@ULVA73B9P> im running prefect in distributed way...

Shubham

08/08/2026, 1:13 PM

@Marvin im running prefect in distributed way on same vm with separate api server, nginx, background service, postgres, redis. im not feeling api is good responsive with all api calls many api calls not working getting timedout while other apis working all the time. curl --location '192.168.29.151/api/work_pools' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YXN0cmE6TmV0d29yazU3MTQ=' \ --data '{ "name": "test", "type": "Docker" }' timedout while health, deployment, get workpool apis working services: postgres: image: postgres:15 container_name: prefect-postgres restart: unless-stopped environment: POSTGRES_USER: prefect POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} POSTGRES_DB: prefect volumes: - postgres_data:/var/lib/postgresql/data healthcheck: test: pg_isready -h localhost -U $$POSTGRES_USER interval: 2s timeout: 5s retries: 15 ports: - "127.0.0.154325432" # bind to localhost only; open to other VMs later when you move DB off-box redis: image: redis:7 container_name: prefect-redis restart: unless-stopped volumes: - redis_data:/data # Runs migrations once, then exits. API/services containers start only after this succeeds. migrate: image: prefecthq/prefect:3-latest container_name: prefect-migrate depends_on: postgres: condition: service_healthy command: prefect server database upgrade -y environment: PREFECT_SERVER_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:${POSTGRES_PASSWORD}@postgres:5432/prefect prefect-api: image: prefecthq/prefect:3-latest container_name: prefect-api restart: unless-stopped depends_on: migrate: condition: service_completed_successfully postgres: condition: service_healthy redis: condition: service_started # --no-services: this instance only serves API/UI traffic; scheduling # and automations run on the separate prefect-services container below. command: prefect server start --host 0.0.0.0 --port 4200 --no-services --keep-alive-timeout 60 environment: PREFECT_HOME: /home/prefect/.prefect PREFECT_SERVER_LOGGING_LEVEL: INFO PREFECT_SERVER_API_AUTH_STRING: ${PREFECT_SERVER_API_AUTH_STRING} # Tells the browser-side UI where to reach the API. Must be an address # your browser (not the container) can resolve — the VM's real IP, # not 127.0.0.1 or a container name. PREFECT_SERVER_UI_API_URL: http://${PREFECT_VM_IP}:4200/api PREFECT_SERVER_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:${POSTGRES_PASSWORD}@postgres:5432/prefect PREFECT_SERVER_DATABASE_MIGRATE_ON_START: "false" PREFECT_SERVER_EVENTS_MESSAGING_BROKER: prefect_redis.messaging PREFECT_SERVER_EVENTS_MESSAGING_CACHE: prefect_redis.messaging PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage PREFECT_REDIS_MESSAGING_HOST: redis PREFECT_REDIS_MESSAGING_PORT: "6379" # Separate Redis DB (1) from app-level use (0), matching Prefect's docs example PREFECT_SERVER_DOCKET_URL: redis://redis:6379/1 volumes: - prefect_home:/home/prefect/.prefect expose: - "4200" prefect-services: image: prefecthq/prefect:3-latest container_name: prefect-services restart: unless-stopped depends_on: migrate: condition: service_completed_successfully postgres: condition: service_healthy redis: condition: service_started command: prefect server services start environment: PREFECT_HOME: /home/prefect/.prefect PREFECT_SERVER_LOGGING_LEVEL: INFO PREFECT_SERVER_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:${POSTGRES_PASSWORD}@postgres:5432/prefect PREFECT_SERVER_DATABASE_MIGRATE_ON_START: "false" PREFECT_SERVER_EVENTS_MESSAGING_BROKER: prefect_redis.messaging PREFECT_SERVER_EVENTS_MESSAGING_CACHE: prefect_redis.messaging PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage PREFECT_REDIS_MESSAGING_HOST: redis PREFECT_REDIS_MESSAGING_PORT: "6379" PREFECT_SERVER_DOCKET_URL: redis://redis:6379/1 nginx: image: nginx:1.27-alpine container_name: prefect-nginx restart: unless-stopped depends_on: - prefect-api ports: - "4200:4200" # this is what workers / UI / API clients hit volumes: - ./nginx.conf/etc/nginx/conf.d/default.confro volumes: postgres_data: redis_data: prefect_home:

Marvin

08/08/2026, 1:13 PM

✅ thought for 327.1 seconds

Marvin

08/08/2026, 1:18 PM

A few things stand out here. Most likely issue: your

POST /api/work_pools

is doing more than the endpoints that work, and in your setup it can block on backend dependencies like Redis / DB / proxy.

GET

endpoints like health or listing work pools are much simpler, so it makes sense they keep working while create calls time out. Also, I notice a likely payload issue:

Copy code

json
{
  "name": "test",
  "type": "Docker"
}

In Prefect 3, work pool type is usually lowercase like

docker

kubernetes

process

, etc. I would try:

Copy code

bash
curl --location '<http://192.168.29.151:4200/api/work_pools/>' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YXN0cmE6TmV0d29yazU3MTQ=' \
--data '{
  "name": "test",
  "type": "docker"
}'

That said, a bad type should normally return an error, not hang, so I don’t think that’s the main problem. What I think is more important in your compose: 1. Redis-backed eventing is enabled 2.

POST /api/work_pools

emits events during creation 3. If Redis is slow/unreachable/misconfigured, that request can hang while simpler GETs still work I also see your compose has some suspicious env formatting in the DB and Redis URLs. In your pasted config, these appear malformed:

Copy code

yaml
PREFECT_SERVER_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect>:${POSTGRES_PASSWORD}@postgres:5432/prefect
PREFECT_SERVER_DOCKET_URL: <redis://redis:6379/1>

Those angle brackets should not be there. They should be plain URLs:

Copy code

yaml
PREFECT_SERVER_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:${POSTGRES_PASSWORD}@postgres:5432/prefect>
PREFECT_SERVER_DOCKET_URL: <redis://redis:6379/1>

If those brackets are really in your actual compose file, fix that first. Recommended checks 1. Inspect API logs while sending the failing request Run:

Copy code

bash
docker logs -f prefect-api

Then send the

POST /api/work_pools

request again and look for: - Redis connection errors - DB timeout errors - stack traces around workers / work pools - auth / proxy issues Also check:

Copy code

bash
docker logs -f prefect-services
docker logs -f prefect-redis
docker logs -f prefect-postgres

2. Verify Redis is reachable from the API container

Copy code

bash
docker exec -it prefect-api sh

Then from inside:

Copy code

bash
python -c "import socket; s=socket.create_connection(('redis',6379),5); print('ok')"

If that hangs or fails, that’s probably your root cause. 3. Verify Postgres is reachable from the API container

Copy code

bash
docker exec -it prefect-api sh
python -c "import socket; s=socket.create_connection(('postgres',5432),5); print('ok')"

4. Fix timeout behavior for Redis Prefect’s Redis messaging can wait too long if socket timeouts are not set. Add these to both

prefect-api

and `prefect-services`:

Copy code

yaml
PREFECT_REDIS_MESSAGING_SOCKET_TIMEOUT: "5"
PREFECT_REDIS_MESSAGING_SOCKET_CONNECT_TIMEOUT: "3"

5. Check nginx timeouts If nginx is in front of the API, make sure it is not cutting off upstream requests too aggressively. Your nginx config should include something like:

Copy code

nginx
location / {
    proxy_pass <http://prefect-api:4200;>
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_connect_timeout 10s;
    proxy_send_timeout 30s;
    proxy_read_timeout 30s;
}

If you want, paste your

nginx.conf

too — that may reveal the problem quickly. 6. Try hitting Prefect API directly, bypassing nginx This is important to isolate whether nginx is the problem. From the VM:

Copy code

bash
curl -v <http://127.0.0.1:4200/api/health>

And if possible, expose/test the container directly without nginx and retry the create work pool request. If direct works but nginx path times out, then the issue is nginx/auth config, not Prefect. 7. Use trailing slash on the endpoint Try: ```bash

Marvin

08/08/2026, 1:18 PM

POST /api/work_pools/

Copy code

instead of:

```bash
POST /api/work_pools

Usually FastAPI handles redirects, but behind nginx/auth sometimes redirect handling can behave oddly. Suggested compose fixes I’d update the relevant env section like this:

Copy code

yaml
environment:
  PREFECT_HOME: /home/prefect/.prefect
  PREFECT_SERVER_LOGGING_LEVEL: INFO
  PREFECT_SERVER_API_AUTH_STRING: ${PREFECT_SERVER_API_AUTH_STRING}
  PREFECT_SERVER_UI_API_URL: http://${PREFECT_VM_IP}:4200/api
  PREFECT_SERVER_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:${POSTGRES_PASSWORD}@postgres:5432/prefect>
  PREFECT_SERVER_DATABASE_MIGRATE_ON_START: "false"
  PREFECT_SERVER_EVENTS_MESSAGING_BROKER: prefect_redis.messaging
  PREFECT_SERVER_EVENTS_MESSAGING_CACHE: prefect_redis.messaging
  PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering
  PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage
  PREFECT_REDIS_MESSAGING_HOST: redis
  PREFECT_REDIS_MESSAGING_PORT: "6379"
  PREFECT_REDIS_MESSAGING_SOCKET_TIMEOUT: "5"
  PREFECT_REDIS_MESSAGING_SOCKET_CONNECT_TIMEOUT: "3"
  PREFECT_SERVER_DOCKET_URL: <redis://redis:6379/1>

And same Redis/DB fixes in

prefect-services

. Most likely root causes in order 1. malformed DB/Redis URLs in env 2. Redis connectivity / event publishing hang 3. nginx proxy timeout / auth interaction 4. request path / redirect issue 5. less likely: work pool type casing Docs / references - Prefect self-hosting docs: https://docs.prefect.io/latest/manage/self-host/ - Prefect workers and work pools: https://docs.prefect.io/latest/deploy/infrastructure-concepts/work-pools/ - Prefect repo: PrefectHQ/prefect If you want, paste: - your actual

nginx.conf

docker logs prefect-api

during one failed request - output of

docker exec prefect-api env | grep PREFECT

and I can narrow it down much more precisely.

Shubham

08/08/2026, 1:21 PM

@Marvin also why everytime new access to ui /api is slow like loading first call fails secondary works all the time . can't we keep our listeniners alive all the time. 131945.646 | INFO | uvicorn.access - 172.18.0.6:55070 - "POST /api/flow_runs/count HTTP/1.0" 200 131946.917 | INFO | uvicorn.access - 172.18.0.6:55076 - "POST /api/work_pools/DRX%20Cloud/get_scheduled_flow_runs HTTP/1.0" 200 131959.279 | INFO | uvicorn.access - 172.18.0.6:42396 - "POST /api/work_pools/DRX%20Cloud/get_scheduled_flow_runs HTTP/1.0" 200 132005.321 | INFO | uvicorn.access - 172.18.0.6:42406 - "POST /api/work_pools HTTP/1.0" 307 132008.528 | INFO | uvicorn.access - 172.18.0.6:39496 - "POST /api/work_pools/DRX%20Cloud/get_scheduled_flow_runs HTTP/1.0" 200 132014.569 | INFO | uvicorn.access - 172.18.0.6:39510 - "POST /api/flow_runs/count HTTP/1.0" 200 132014.571 | INFO | uvicorn.access - 172.18.0.6:39498 - "POST /api/flow_runs/count HTTP/1.0" 200 upstream prefect_api { least_conn; server prefect-api:4200 max_fails=3 fail_timeout=30s; # server <vm2-ip>:4200 max_fails=3 fail_timeout=30s; # <- uncomment & fill in when VM2 exists } server { listen 4200; # Fast liveness check path, low timeouts so a hung instance is dropped quickly location /api/health { proxy_pass prefect_api; proxy_connect_timeout 1s; proxy_read_timeout 1s; } location /api { proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; # WebSocket support (required for live UI updates) proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; # Pass through the basic-auth header used by PREFECT_SERVER_API_AUTH_STRING proxy_set_header Authorization $http_authorization; proxy_pass_header Authorization; proxy_pass prefect_api; } location / { proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_pass prefect_api; } } root@Prefect:/home/prefect/prefect

Marvin

08/08/2026, 1:21 PM

✅ thought for 306.6 seconds

Marvin

08/08/2026, 1:26 PM

Yes — your nginx config is the main reason for the “first call slow/fails, second works” behavior. Your logs already show two important clues: 1.

HTTP/1.0

is being sent from nginx to Prefect 2.

POST /api/work_pools HTTP/1.0" 307

That combination is a big red flag. What’s happening ### 1. nginx is proxying to Prefect with HTTP/1.0 By default, nginx talks to upstreams using HTTP/1.0 unless you set:

Copy code

nginx
proxy_http_version 1.1;

Without that: - connections are not reused properly - keepalive behavior is poor - POST/body handling is less reliable - first request often pays connection setup cost - some requests get redirected/retried oddly That is why you see this in uvicorn:

Copy code

text
"POST /api/work_pools HTTP/1.0" 307

### 2. your

/api

block sets

Connection: upgrade

for every API request This is wrong for normal REST calls. You currently have:

Copy code

nginx
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

inside the general

/api

location. That should only be used for WebSocket endpoints, not all API routes. For normal REST requests, forcing

Connection: upgrade

can interfere with connection reuse and produce flaky behavior. ### 3. your

/api/work_pools

request is getting a

That usually means redirect behavior due to path normalization, commonly: -

/api/work_pools

/api/work_pools/

- combined with proxy behavior / HTTP/1.0 So nginx sends request one way, upstream responds with redirect, client retries, second time works. That matches your symptom exactly. --- Can listeners be kept alive all the time? Yes — but it’s mostly nginx upstream keepalive and HTTP/1.1 that matter here, not just Prefect’s

--keep-alive-timeout

. You already set:

Copy code

bash
prefect server start --host 0.0.0.0 --port 4200 --no-services --keep-alive-timeout 60

That helps on the Prefect side, but nginx is still talking HTTP/1.0 unless you explicitly change it. So Prefect is ready for keepalive, but nginx is not taking advantage of it. --- What to change Here is a much better nginx config for your setup:

Copy code

nginx
upstream prefect_api {
    least_conn;
    server prefect-api:4200 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

server {
    listen 4200;

    location /api/health {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass <http://prefect_api>;
        proxy_connect_timeout 1s;
        proxy_read_timeout 1s;
    }

    location /api/events {
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_set_header Authorization $http_authorization;
        proxy_pass_header Authorization;

        proxy_read_timeout 3600s;
        proxy_pass <http://prefect_api>;
    }

    location /api/ {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;

        proxy_set_header Authorization $http_authorization;
        proxy_pass_header Authorization;

        proxy_connect_timeout 10s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;

        proxy_pass <http://prefect_api>;
    }

    location / {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;

        proxy_connect_timeout 10s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;

        proxy_pass <http://prefect_api>;
    }
}

--- Why this fixes it ###

proxy_http_version 1.1;

This makes nginx talk to Prefect using HTTP/1.1 instead of HTTP/1.0.

Marvin

08/08/2026, 1:26 PM

That gives you: - better connection reuse - fewer first-request delays - cleaner POST handling - less weird redirect/retry behavior ###

keepalive 32;

This keeps upstream connections warm between nginx and Prefect. So nginx doesn’t need to create a new upstream TCP connection for every request. ###

proxy_set_header Connection "";

This is important for normal REST routes. It avoids forcing upgrade semantics on every API call. ### separate

/api/events

If Prefect UI uses live updates/websocket-style event connections, those are the only routes that should get:

Copy code

nginx
Connection "upgrade"
Upgrade $http_upgrade

not all

/api

requests. --- Also fix the trailing slash behavior Your log shows:

Copy code

text
POST /api/work_pools HTTP/1.0" 307

Try calling:

Copy code

bash
curl --location '<http://192.168.29.151:4200/api/work_pools/>' ...

with the trailing slash. That may stop the redirect immediately. --- One more improvement Change this block:

Copy code

nginx
location /api {

to:

Copy code

nginx
location /api/ {

This is safer and more predictable for API routing. --- What I suspect is happening right now Current flow is likely: 1. browser/curl hits nginx 2. nginx opens fresh upstream connection using HTTP/1.0 3. nginx sends POST

/api/work_pools

4. upstream responds with 307 redirect to normalized path 5. first attempt feels slow/fails 6. retry or second request succeeds So yes, this is very consistent with your symptoms. --- Important note on Prefect itself This is probably not because Prefect listeners are “sleeping” or “not alive.” Your Prefect API process is alive. The issue is more: - reverse proxy connection handling - path redirect - upstream HTTP version - possible Redis blocking for write endpoints --- Recommended next steps 1. Update nginx config as above 2. Reload nginx 3. Retry using

/api/work_pools/

with trailing slash 4. Watch logs again You want to see

HTTP/1.1

in uvicorn logs instead of

HTTP/1.0

. --- How to reload nginx

Copy code

bash
docker exec prefect-nginx nginx -t
docker exec prefect-nginx nginx -s reload

--- What success should look like After the fix, uvicorn logs should look more like:

Copy code

text
"POST /api/work_pools/ HTTP/1.1" 201

instead of:

Copy code

text
"POST /api/work_pools HTTP/1.0" 307

--- Relevant references - Prefect self-hosting: https://docs.prefect.io/latest/manage/self-host/ - Prefect work pools: https://docs.prefect.io/latest/deploy/infrastructure-concepts/work-pools/ - nginx proxy docs: nginx proxy module If you want, I can rewrite your full

docker-compose.yml

and

nginx.conf

into a cleaned-up working version for this architecture.

Shubham

08/08/2026, 2:20 PM

mar

Shubham

08/08/2026, 2:20 PM

@Marvin from postman prefect api call working but from my ui : 142017.487 | INFO | uvicorn.access - 172.18.0.6:43666 - "POST /api/work_pools HTTP/1.1" 307

Marvin

08/08/2026, 2:20 PM

✅ thought for 90.4 seconds

Marvin

08/08/2026, 2:22 PM

That narrows it down a lot. If Postman works but the UI gets:

Copy code

text
POST /api/work_pools HTTP/1.1" 307

then the problem is probably not Redis or Postgres for this specific symptom. It points much more strongly to: the UI is calling the non-canonical path

/api/work_pools

and Prefect is redirecting it to

/api/work_pools/

Postman follows redirects cleanly, but the browser/UI flow may not handle that redirect well for this request. Why this happens In FastAPI/Starlette, routes defined with a trailing slash often redirect: - request to

/api/work_pools

- server returns

307 Temporary Redirect

- correct path is

/api/work_pools/

preserves method/body, which is why it is used for POST. So your UI is likely making a request to the non-slash version and getting redirected. Why the UI fails but Postman works - Postman usually follows redirects more predictably - browser fetch/XHR behavior can be affected by: - auth headers on redirect - proxy behavior - CORS / mixed handling - frontend code not expecting redirected POSTs Since your log now shows

HTTP/1.1

, your earlier nginx HTTP/1.0 issue may be improved, but this 307 redirect is still the remaining problem. Most likely fix Make nginx normalize

/api/work_pools

/api/work_pools/

before it reaches Prefect, or make sure the UI is configured to call the canonical path. Because you cannot easily change the built UI behavior, nginx redirect handling is usually the practical fix. You can add a specific rewrite:

Copy code

nginx
location = /api/work_pools {
    return 308 /api/work_pools/;
}

or proxy it directly in a normalized way. But before doing that, I’d first make your API block use the slash form:

Copy code

nginx
location /api/ {
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;

    proxy_set_header Authorization $http_authorization;
    proxy_pass_header Authorization;

    proxy_connect_timeout 10s;
    proxy_send_timeout 60s;
    proxy_read_timeout 60s;

    proxy_pass <http://prefect_api>;
}

Note the block is

location /api/

not

location /api

. Also important: your

PREFECT_SERVER_UI_API_URL

should be exactly correct. You currently have:

Copy code

yaml
PREFECT_SERVER_UI_API_URL: http://${PREFECT_VM_IP}:4200/api

That is usually right, but if the UI builds requests by appending paths, the slash handling matters. I would try:

Copy code

yaml
PREFECT_SERVER_UI_API_URL: http://${PREFECT_VM_IP}:4200/api/

If Prefect accepts it cleanly in your version, this can help prevent path concatenation oddities. If not, keep it without trailing slash — but it’s worth checking. Best immediate test Open browser dev tools → Network tab and inspect the failing request. Check: 1. request URL exactly 2. response headers on the

Location

header value 4. whether the follow-up redirected request is sent 5. whether

Authorization

header is preserved on redirect I strongly suspect you’ll see:

Copy code

text
Location: /api/work_pools/

If so, that confirms this is a redirect/canonical-path problem. Recommended nginx addition Try this: ```nginx upstream prefect_api { least_conn; server prefect-api:4200 max_fails=3 fail_timeout=30s; keepalive 32; } server { listen 4200; location = /api/work_pools { return 308 /api/work_pools/; } location /api/health { proxy_http_version 1.1; proxy_set_header Connection ""; proxy_pass http://prefect_api; proxy_connect_timeout 1s; proxy_read_timeout 1s; } location /api/events { proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Upgrade $http_upgrade;

Marvin

08/08/2026, 2:22 PM

proxy_set_header Connection "upgrade"; proxy_set_header Authorization $http_authorization; proxy_pass_header Authorization; proxy_read_timeout 3600s; proxy_pass http://prefect_api; } location /api/ { proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Authorization $http_authorization; proxy_pass_header Authorization; proxy_connect_timeout 10s; proxy_send_timeout 60s; proxy_read_timeout 60s; proxy_pass http://prefect_api; } location / { proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://prefect_api; } }

Copy code

Then reload nginx:

```bash
docker exec prefect-nginx nginx -t
docker exec prefect-nginx nginx -s reload

One caution A

from nginx still redirects. If the browser-side issue is “POST redirects are handled badly”, then even that may not be enough. In that case, the better approach is to let the request hit the exact upstream path the UI expects, or verify whether the Prefect UI/API version combination is mismatched. So I’d also want to know: - what Prefect image tag are you using exactly? (

3-latest

can drift) - does the browser network tab show redirect follow-up or not? - what is the

Location

header in the 307? If you share the browser network details for that request, I can tell you the exact fix.

Shubham

08/08/2026, 2:34 PM

@Marvin im still feeling delay in api responses im getting frustrated after every few minutes idl it failing first attempt and second working

Marvin

08/08/2026, 2:34 PM

✅ thought for 22.9 seconds

Marvin

08/08/2026, 2:34 PM

I get why you’re frustrated. At this point, this does not sound like a simple “wrong endpoint” problem anymore — it sounds like an intermittent infrastructure/proxy issue. Since: - Postman works - UI sometimes fails first, then works second - earlier logs showed HTTP/1.0 and redirects - you still feel periodic delay every few minutes the most likely remaining causes are: 1. nginx upstream connection reuse is bad/stale 2. Prefect API container is briefly blocking on Redis/DB for write-ish requests 3. using
3-latest
may have introduced moving-version mismatch behavior between UI/API assets over time --- ## What I would do next — simplest high-confidence fixes ### A. Stop using

3-latest

Pin an exact version for all Prefect containers. For example, use the same tag everywhere:

Copy code

yaml
image: prefecthq/prefect:3.8.2

for: -

migrate

prefect-api

prefect-services

Using

3-latest

can make troubleshooting miserable. You can see recent release notes here: Prefect releases --- ### B. Replace nginx config with a clean known-good version Use this exactly:

Copy code

nginx
upstream prefect_api {
    least_conn;
    server prefect-api:4200 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

server {
    listen 4200;

    location /api/health {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass <http://prefect_api>;
        proxy_connect_timeout 1s;
        proxy_read_timeout 1s;
    }

    location /api/events {
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Authorization $http_authorization;
        proxy_pass_header Authorization;
        proxy_read_timeout 3600s;
        proxy_pass <http://prefect_api>;
    }

    location /api/ {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Authorization $http_authorization;
        proxy_pass_header Authorization;
        proxy_connect_timeout 10s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
        proxy_pass <http://prefect_api>;
    }

    location / {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_connect_timeout 10s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
        proxy_pass <http://prefect_api>;
    }
}

Then reload nginx:

Copy code

bash
docker exec prefect-nginx nginx -t
docker exec prefect-nginx nginx -s reload

--- ### C. Fix your Prefect env URLs if they really contain brackets These should be plain values:

Copy code

yaml
PREFECT_SERVER_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:${POSTGRES_PASSWORD}@postgres:5432/prefect>
PREFECT_SERVER_DOCKET_URL: <redis://redis:6379/1>

If the

<...>

is only formatting in Slack, ignore this. If it is in the real file, fix immediately. --- ### D. Add Redis socket timeouts To both

prefect-api

and `prefect-services`:

Copy code

yaml
PREFECT_REDIS_MESSAGING_SOCKET_TIMEOUT: "5"
PREFECT_REDIS_MESSAGING_SOCKET_CONNECT_TIMEOUT: "3"

This helps prevent long blocking behavior on event publishing. --- ## Best isolation test This is the fastest way to stop guessing: ### Test 1: bypass nginx completely From the VM, hit Prefect directly. If nginx is removed from the path and the problem disappears, then nginx is the culprit.

Marvin

08/08/2026, 2:34 PM

If your api container is not directly published, temporarily expose it for testing, or exec from inside nginx container / host network path. For example, from inside nginx container:

Copy code

bash
docker exec -it prefect-nginx sh
apk add --no-cache curl
curl -v <http://prefect-api:4200/api/health>

And test the slow endpoint too. ### Test 2: repeatedly hit API and measure latency Run this from the VM against nginx:

Copy code

bash
for i in $(seq 1 20); do
  date
  curl -s -o /dev/null -w "code=%{http_code} connect=%{time_connect} starttransfer=%{time_starttransfer} total=%{time_total}\n" \
    <http://127.0.0.1:4200/api/health>
  sleep 15
done

Then compare against direct upstream:

Copy code

bash
for i in $(seq 1 20); do
  date
  curl -s -o /dev/null -w "code=%{http_code} connect=%{time_connect} starttransfer=%{time_starttransfer} total=%{time_total}\n" \
    <http://prefect-api:4200/api/health>
  sleep 15
done

If nginx path shows spikes and direct path does not, problem is nginx. --- ## Also check container resource pressure Intermittent “first one fails after idle” can also happen if the VM is under pressure. Check:

Copy code

bash
docker stats

and on host:

Copy code

bash
top
free -m
vmstat 1 10

Look for: - high CPU - memory pressure - swap activity - postgres or redis getting stuck --- ## Check logs at the same time Run all of these in parallel while reproducing:

Copy code

bash
docker logs -f prefect-api
docker logs -f prefect-services
docker logs -f prefect-nginx
docker logs -f prefect-postgres
docker logs -f prefect-redis

What matters is whether the delay aligns with: - nginx upstream timeout / reconnect - Redis stall - Postgres wait - API worker pause --- ## My honest read Based on everything you’ve shared, nginx is still the #1 suspect. Especially because: - you had HTTP/1.0 upstream traffic - you had broad

Connection: upgrade

- the issue happens after idle periods - second try works - Postman can work while browser/UI is flaky That pattern is classic reverse-proxy connection handling. --- ## If you want the fastest path I’d recommend this order: 1. pin Prefect to

3.8.2

2. replace nginx config with the clean version above 3. add Redis socket timeouts 4. restart all containers 5. test direct-to-api vs nginx path --- ## Restart sequence After updating config:

Copy code

bash
docker compose down
docker compose up -d

Then verify:

Copy code

bash
docker compose ps
docker logs prefect-api --tail 100
docker logs prefect-nginx --tail 100

--- ## If you want, send me these 3 things next 1. output of

docker compose ps

2. output of

docker stats --no-stream

3. browser network screenshot/details for one failed first request And I can help pinpoint whether this is nginx, Redis, or resource contention.

Shubham

08/08/2026, 2:45 PM

@Marvin intermittent non responding attempts don't even see any log on nginx ?

Marvin

08/08/2026, 2:45 PM

✅ thought for 18.9 seconds

Marvin

08/08/2026, 2:45 PM

Yes — that is a very important clue. If a request does not show up in nginx logs at all, then the failure is happening before nginx handles the request. That means the intermittent non-response is likely in one of these layers: 1. browser/UI side 2. TCP connection establishment to nginx 3. host networking / Docker port forwarding 4. client waiting on a stale keepalive connection before retrying Not seeing anything in nginx strongly suggests: the request never reached nginx application processing. --- ## What this usually means ### Case 1: browser is reusing a stale connection This is very common. The browser keeps an idle connection open to

192.168.29.151:4200

. After some idle time, that connection is dead or half-closed somewhere in the chain: - browser thinks it is reusable - nginx/docker/host has already dropped it - first request on that stale socket hangs/fails - browser opens a new connection - second attempt works This fits your symptom very well: - after a few minutes idle - first attempt fails - second works - nginx may not log the failed one because request bytes never fully arrive --- ## Why nginx may have no log nginx access logs are typically written only when it actually receives/processes the HTTP request. If the problem is: - stale TCP socket - client-side socket reuse problem - connection reset before request line arrives - Docker NAT / bridge issue before HTTP is parsed then nginx access log may stay empty. --- ## Most likely suspects now ### 1. stale keepalive connection between browser and nginx This is my top suspect now. ### 2. Docker port publishing / bridge networking oddity You expose nginx with:

Copy code

yaml
ports:
  - "4200:4200"

Sometimes intermittent first-hit failures come from Docker networking/conntrack behavior, especially on busy VMs. ### 3. browser-side fetch/request timeout/retry behavior Possible if the UI is trying to reuse a dead connection or cached session state. ### 4. host firewall / conntrack / reverse path issues Less likely, but possible. --- ## What to do next ## A. Enable detailed nginx logging Right now, we need to prove whether requests are reaching nginx at all. In nginx config, add:

Copy code

nginx
log_format detailed '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    'rt=$request_time ua="$upstream_addr" '
                    'uct="$upstream_connect_time" '
                    'uht="$upstream_header_time" '
                    'urt="$upstream_response_time"';

access_log /var/log/nginx/access.log detailed;
error_log /var/log/nginx/error.log info;

Then reload nginx:

Copy code

bash
docker exec prefect-nginx nginx -t
docker exec prefect-nginx nginx -s reload

Then tail logs:

Copy code

bash
docker exec -it prefect-nginx sh
tail -f /var/log/nginx/access.log /var/log/nginx/error.log

If failed attempts still show nothing, that confirms pre-nginx failure. --- ## B. Reduce browser→nginx keepalive lifetime If stale client connections are the issue, reducing keepalive timeout can help. Inside

server

block:

Copy code

nginx
keepalive_timeout 15s;
keepalive_requests 100;

This tells nginx not to hold browser connections too long. You can even test aggressively:

Copy code

nginx
keepalive_timeout 5s;

If that improves “first request after idle” behavior, stale keepalive was the problem. --- ## C. Disable keepalive temporarily to test For debugging only, try:

Copy code

nginx
keepalive_timeout 0;

If the intermittent first-failure disappears, then you’ve confirmed client-side stale connection reuse. --- ## D. Test with curl repeatedly from another machine Run this from the machine where you access the UI:

Copy code

bash
for i in $(seq 1 20); do
  date
  curl -v --max-time 10 <http://192.168.29.151:4200/api/health>
  sleep 30
done

Watch whether: - failed attempts happen after idle - failures appear before nginx logs

Marvin

08/08/2026, 2:45 PM

- second retry succeeds immediately If yes, that strongly points to stale TCP connection / keepalive reuse. --- ## E. Test host locally vs remote client From the VM itself:

Copy code

bash
curl -v <http://127.0.0.1:4200/api/health>

From your laptop/client:

Copy code

bash
curl -v <http://192.168.29.151:4200/api/health>

If local is always fast but remote has first-hit failures, then: - nginx itself is probably okay - issue is network path / browser socket reuse / host port exposure --- ## F. Check Docker / kernel conntrack pressure On the VM:

Copy code

bash
dmesg | tail -100
sysctl net.netfilter.nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count

If conntrack is near max, new connections can behave badly. Also check:

Copy code

bash
ss -tan | grep :4200 | head -50

Look for lots of

TIME-WAIT

CLOSE-WAIT

, or stale states. --- ## G. Try bypassing Docker port publishing If possible, test nginx on host networking or expose Prefect directly for comparison. For example, if direct access to Prefect container is stable but docker-published nginx is not, the issue may be Docker bridge/NAT. --- ## H. Browser devtools check In the browser network tab, inspect one failed request: - does it show

stalled

? -

pending

? -

failed

with no response headers? -

ERR_EMPTY_RESPONSE

? -

ERR_CONNECTION_RESET

? -

ERR_NETWORK_CHANGED

? If there are no response headers at all, that again means HTTP never reached nginx. --- ## My best guess now Because you said:

intermittent non responding attempts don't even see any log on nginx

the leading explanation is: the browser/client is attempting to reuse an old dead TCP connection to nginx, the first request dies before HTTP is parsed, and the second attempt opens a fresh socket and works. That is consistent with: - happens after idle - second works - no nginx access log - earlier keepalive/proxy weirdness --- ## Fastest test to confirm Add this to nginx:

Copy code

nginx
server {
    listen 4200;
    keepalive_timeout 5s;
    keepalive_requests 100;
    ...
}

Reload nginx and test again. If the issue becomes much less frequent, that’s your answer. --- ## If you want, I can give you: 1. a full debug nginx.conf with logging and safer keepalive settings, or 2. a clean production nginx.conf optimized for Prefect UI/API. If you want, I’d suggest I generate the debug version first.

Shubham

08/08/2026, 4:29 PM

Marvin

08/08/2026, 4:29 PM

Shubham

08/08/2026, 4:41 PM

Marvin

08/08/2026, 4:41 PM

✅ thought for 9.7 seconds

Marvin

08/08/2026, 4:41 PM

Yes — those WebSocket log lines are normal for the Prefect UI. The UI opens event connections like: -

/api/events/in

/api/events/out

So logs like this are expected:

Copy code

text
"WebSocket /api/events/in" [accepted]
"WebSocket /api/events/out" [accepted]
connection open

That means the UI is connecting to Prefect’s event streams. --- ## But one thing is not ideal You’re seeing many repeated WebSocket opens:

Copy code

text
16:17:12 ...
16:17:16 ...
16:17:20 ...
16:17:38 ...

A few opens are fine, but frequent reconnects can indicate: - browser tab reconnecting repeatedly - nginx WebSocket proxy config not stable - idle connection being dropped - UI retry loop because socket closes unexpectedly So the log lines themselves are normal, but the frequency may be a symptom. --- ## What should happen normally Normally: - UI loads - opens websocket/event connections - keeps them open for a while Not: - reconnect every few seconds or very often If they reconnect too often, that can contribute to the “first attempt slow/fails” feeling in the UI. --- ## Your nginx should treat

/api/events

separately This part is important. Use a dedicated block like:

Copy code

nginx
location /api/events {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Authorization $http_authorization;
    proxy_pass_header Authorization;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
    proxy_pass <http://prefect_api>;
}

And for normal API:

Copy code

nginx
location /api/ {
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Authorization $http_authorization;
    proxy_pass_header Authorization;
    proxy_connect_timeout 10s;
    proxy_send_timeout 60s;
    proxy_read_timeout 60s;
    proxy_pass <http://prefect_api>;
}

--- ## What to look for next Check whether you also see: -

connection closed

- websocket disconnect errors - 101 switching protocol responses in nginx logs - repeated browser reconnects in devtools If these sockets are reconnecting every few seconds, then nginx/browser connection stability is still an issue. --- ## Short answer - Yes, those websocket accepts are normal - No, lots of repeated reconnects may not be normal - they could be part of the UI instability you’re seeing If you want, I can give you a final cleaned nginx.conf specifically for Prefect UI + API + WebSockets so you can swap it in directly.

4 Views

Open in Slack

Previous Next