<@ULVA73B9P> My workers will randomly die and the ...
# ask-marvin
m
@Marvin My workers will randomly die and the only error I can see in my logs is this: [Prefect Error] 015213.373 | ERROR | prefect.workers.process.worker_dev - Worker has not polled in the last 307 seconds and should be restarted
m
thought for 126.3 seconds
That log line usually means the worker’s polling loop stopped making progress for long enough that Prefect marked it unhealthy. A few important details: - The message is tied to the worker healthcheck logic - For workers, Prefect tracks the last successful poll for scheduled work - If it hasn’t polled in roughly
PREFECT_WORKER_QUERY_SECONDS * 30
, it logs:
Worker has not polled in the last ... seconds and should be restarted
- With the default poll interval of
10s
, the threshold is about 300 seconds, so your
307 seconds
is right at that cutoff So this is usually a symptom, not the root cause. Common causes - Temporary or repeated loss of connectivity to your Prefect API / Cloud - API timeouts / transport errors / 5xx responses - The worker process getting stuck or starved for CPU/memory - Very slow environment / host pauses - Less commonly, auth/config issues with
PREFECT_API_URL
or
PREFECT_API_KEY
What I’d check first 1. Turn up worker logging Restart the worker with debug logging:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG PREFECT_WORKER_DEBUG_MODE=true prefect worker start --pool "your-pool"
Then look before the
has not polled
line for things like: -
TransportError
-
HTTPStatusError
- retries / backoff messages - connection reset / DNS / timeout errors 2. Verify API connectivity from the worker host From the same machine/container running the worker, confirm it can consistently reach your API. 3. Check host/container health If the worker host is under memory pressure, CPU throttling, or being paused/restarted by your platform, the polling loop can miss its deadline. 4. If you are using healthchecks/liveness probes A failing
/health
endpoint may cause Kubernetes/Docker/systemd to restart the worker, which is expected behavior once the worker is considered unhealthy. Mitigations - If your environment is just slow/latent, try increasing the worker poll interval:
Copy code
export PREFECT_WORKER_QUERY_SECONDS=30
This also increases the unhealthy threshold to about
900s
because the threshold is based on that value. - Make sure the worker has stable outbound network access to Prefect - If running in Kubernetes, check for: - pod evictions - CPU throttling - OOM kills - node instability Healthcheck flag If you’re explicitly using worker healthchecks, the CLI option is:
Copy code
prefect worker start --pool "your-pool" --with-healthcheck
I verified the command exists: - Prefect docs - Worker healthcheck behavior was introduced here: PR #9687 I also didn’t find a clear existing GitHub issue specifically diagnosing this exact message beyond the healthcheck implementation itself. If you want, I can help you narrow this down further if you share: - where the worker is running (Docker, Kubernetes, VM, ECS, etc.) - whether you’re using Prefect Cloud or self-hosted server - the 20–50 log lines before this error