Thanks for the logs — they point to two separate (but related) problems: 1) Server/services loops are slow - Warnings like: - RecentDeploymentsScheduler took 8s for a 5s loop - MarkLateRuns / FailExpiredPauses took >5s - These mean the server background services aren’t completing their iterations fast enough. On a single host with Postgres (RDS) this is usually due to database latency/connection pooling or resource pressure (CPU/memory). - If the server is under memory pressure, it can get killed — the “Terminated” line strongly suggests an OOM kill or a supervisor stopping the process. 2) Worker cannot reach the API for events - The error: - Unable to connect to 'ws://127.0.0.1:4200/api/events/in' - This happens when the worker’s

PREFECT_API_URL

points at 127.0.0.1 but the API is not actually reachable at that address from the worker’s network namespace. Common case: the worker is in a container and 127.0.0.1 refers to the container itself, not the host/server. - It also occurs if the server is down/restarting — then the worker can’t open the WebSocket and you’ll see “EventsWorker failed to process item … worker.stopped”. Requests to confirm - How are you running the server and the process worker (bare metal or Docker)? What exact commands/compose are you using? - What is the worker’s configured API URL? - Run this where the worker runs:

prefect config view --show-sources | grep api.url

- Are you using any RRule-based schedules (e.g., iCal RRULE)? These can be heavy on the scheduler. - Instance size and memory usage; do you see OOM kills? - On Linux: check

dmesg -T | grep -i oom

Quick checks and fixes A) Make sure the worker can reach the API - If the worker runs on the same EC2 host but in Docker, 127.0.0.1 is not the host. Point it to the host’s reachable address (EC2 private IP, hostname, or

host.docker.internal

on Docker Desktop). - Example:

export PREFECT_API_URL=http://<EC2-private-ip>:4200/api

- Start the server bound to all interfaces so the worker can reach it:

prefect server start --host 0.0.0.0

Confirm health from the worker’s context:

curl -sS http://<EC2-private-ip>:4200/health

- Docs on settings and `PREFECT_API_URL`: - Settings and profiles - Manage settings B) Reduce service load and DB latency sensitivity (RDS) - If you’re on RDS, tune the server’s connection pool and recycle to avoid stale connections and long waits:

export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
  export PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
  export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=600   # 10 minutes
  export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=30
  export PREFECT_SERVER_DATABASE_TIMEOUT=30
  export PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT=10

- If you have RRule schedules, prefer cron/interval where possible — heavy RRULEs can push the scheduler CPU/DB: - Related: Scheduler CPU spikes with RRule - Consider relaxing scheduler loop/limits while you stabilize:

export PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS=60
  export PREFECT_SERVER_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE=50
  export PREFECT_SERVER_SERVICES_SCHEDULER_MAX_RUNS=50

- We’ve seen slow loops and timeouts correlated with DB timeouts/locks or undersized pools in 3.x: - Server timeouts due to slow DB comms - Slower API/UI after upgrade — pool/timeout tuning C) Check for OOM and oversubscription - If the server process is being killed, increase instance memory or reduce concurrency. On a single EC2 host running server + worker(s) + flows, it’s easy to overcommit.

- Your flow uses a ThreadPoolTaskRunner(max_workers=8). With a process worker, each flow run executes in its own process; each process then uses up to 8 threads — multiply by concurrent flow runs and you can run out of RAM quickly. Consider: - Lowering

max_workers

- Reducing concurrent runs in your work pool - Limiting task concurrency via tags if needed D) Enable more logging temporarily - To see the actual connection error for the events WebSocket:

export PREFECT_DEBUG_MODE=1

- Start the server with higher log level if needed:

prefect server start --host 0.0.0.0 --log-level DEBUG

Why these symptoms line up with your logs - The “Terminated” suggests the server process stopped (likely OOM or manual stop). Immediately after, the worker tried to send a “worker.stopped” event and could not connect to ws://127.0.0.1:4200/api/events/in — because the API was no longer listening or was never reachable from the worker’s namespace. - The “took longer than its loop interval” warnings indicate the server was already struggling (DB latency, pool exhaustion, or CPU pressure), making it more likely to be killed by the OOM killer. If you can share: - How you start the server/worker (commands or compose) - Output of

prefect config view --show-sources | grep api.url

from the worker - Instance specs and any OOM logs …I can give you an exact, step-by-step config for your single-EC2 + RDS setup.