< Marvin> I m looking at the logs for my prefect ProcessWork Prefect Community #ask-marvin

<@ULVA73B9P> I'm looking at the logs for my prefec...

Janet Carson

06/01/2026, 4:26 PM

@Marvin I'm looking at the logs for my prefect ProcessWorker work pool. It's happily working along until line 13 -- exiting worker context -- and after that bit of code it never queries for flows again. What causes this? My worker is on prefect 3.7.2 1 sh[523528]: 012201.394 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of 'sync_with_backend' 2 sh[523528]: 012201.722 | DEBUG | prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Worker synchronized with the Prefect API server. Remote ID: 65fc9967-6b48-43c0-8d28-3531f89afa7c 3 sh[523528]: 012206.339 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs' 4 sh[523528]: 012206.339 | DEBUG | prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Querying for flow runs scheduled before 2026-05-30 082216.339813+00:00 5 sh[523528]: 012206.514 | DEBUG | prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Discovered 0 scheduled_flow_runs 6 sh[523528]: 012218.219 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs' 7 sh[523528]: 012218.220 | DEBUG | prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Querying for flow runs scheduled before 2026-05-30 082228.220099+00:00 8 sh[523528]: 012218.384 | DEBUG | prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Discovered 0 scheduled_flow_runs 9 sh[523528]: 012228.056 | DEBUG | prefect.utilities.services.critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 sh[523528]: 012228.057 | DEBUG | prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Querying for flow runs scheduled before 2026-05-30 082238.057320+00:00 11 sh[523528]: 012228.189 | DEBUG | prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Discovered 0 scheduled_flow_runs 12 sh[523528]: 012232.079 | DEBUG | prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Worker channel unhealthy, REST fallback is active: received 1013 (try again later) service_unavailable; then sent 1013 (try again later) service_unavailable 13 sh[523528]: 012232.080 | DEBUG | prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Exiting worker context... 14 sh[523528]: 012232.081 | DEBUG | prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Tearing down worker... 15 sh[523528]: 012232.083 | DEBUG | prefect.runner - Stopping runner... 16 sh[523528]: 012232.083 | DEBUG | prefect.FlowRunCancellingObserver - Shutting down FlowRunCancellingObserver 17 sh[523528]: 012232.084 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Emitting event id=019e77fa-4993-76b9-9b76-bea0068d701a. 18 sh[523528]: 012232.085 | DEBUG | prefect.events.clients - Added event id=019e77fa-4993-76b9-9b76-bea0068d701a to unconfirmed events list. There are now 1 unconfirmed events. 19 sh[523528]: 012232.085 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Emit reconnection attempt 0. 20 sh[523528]: 012232.086 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Sending event id=019e77fa-4993-76b9-9b76-bea0068d701a. 21 sh[523528]: 012232.087 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Got retryable error: ConnectionClosedError 22 sh[523528]: 012232.097 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Emit reconnection attempt 1. 23 sh[523528]: 012232.108 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Attempting websocket reconnection. 24 sh[523528]: 012232.113 | DEBUG | prefect.events.clients - Reconnecting websocket connection. 25 sh[523528]: 012232.119 | DEBUG | prefect.events.clients - Cleared existing websocket connection. 26 sh[523528]: 012232.125 | DEBUG | prefect.events.clients - Opening websocket connection. 27 sh[523528]: 012232.349 | DEBUG | prefect.events.clients - Pinging to ensure websocket connected. 28 sh[523528]: 012232.450 | DEBUG | prefect.events.clients - Pong received. Websocket connected. 29 sh[523528]: 012232.461 | DEBUG | prefect.events.clients - Resending 1 unconfirmed events. 30 sh[523528]: 012232.467 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Emitting event id=019e77fa-4993-76b9-9b76-bea0068d701a. 31 sh[523528]: 012232.472 | DEBUG | prefect.events.clients - Added event id=019e77fa-4993-76b9-9b76-bea0068d701a to unconfirmed events list. There are now 1 unconfirmed events. 32 sh[523528]: 012232.477 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Emit reconnection attempt 0. 33 sh[523528]: 012232.483 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Sending event id=019e77fa-4993-76b9-9b76-bea0068d701a. 34 sh[523528]: 012232.509 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Checkpointing event id=019e77fa-4993-76b9-9b76-bea0068d701a. 35 sh[523528]: 012232.515 | DEBUG | prefect.events.clients - Finished resending unconfirmed events. 36 sh[523528]: 012232.520 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Sending event id=019e77fa-4993-76b9-9b76-bea0068d701a. 37 sh[523528]: 012232.546 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Checkpointing event id=019e77fa-4993-76b9-9b76-bea0068d701a. 38 sh[523528]: 012302.559 | DEBUG | prefect.events.clients - Time-based checkpoint: confirming 1 events. 39 sh[523528]: 012302.665 | DEBUG | prefect.events.clients - EventsClient(id=137765344224272): Pong received. Events checkpointed.

Marvin

06/01/2026, 4:26 PM

Your message was too long, here's your message at the allowed limit: I'm looking at the logs for my prefect ProcessWorker work pool It's happily working along until line 13 -- exiting worker context -- and after that bit of code it never queries for flows again What causes this? My worker is on prefect 3 7 2 1 sh[523528] 01 22 01 394 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'sync_with_backend' 2 sh[523528] 01 22 01 722 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Worker synchronized with the Prefect API server Remote ID 65fc9967-6b48-43c0-8d28-3531f89afa7c 3 sh[523528] 01 22 06 339 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 4 sh[523528] 01 22 06 339 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Querying for flow runs scheduled before 2026-05-30 08 22 16 339813+00 00 5 sh[523528] 01 22 06 514 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Discovered 0 scheduled_flow_runs 6 sh[523528] 01 22 18 219 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 7 sh[523528] 01 22 18 220 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Querying for flow runs scheduled before 2026-05-30 08 22 28 220099+00 00 8 sh[523528] 01 22 18 384 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Discovered 0 scheduled_flow_runs 9 sh[523528] 01 22 28 056 | DEBUG | prefect utilities services critical_service_loop - Starting run of 'get_and_submit_flow_runs' 10 sh[523528] 01 22 28 057 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Querying for flow runs scheduled before 2026-05-30 08 22 38 057320+00 00 11 sh[523528] 01 22 28 189 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Discovered 0 scheduled_flow_runs 12 sh[523528] 01 22 32 079 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Worker channel unhealthy REST fallback is active received 1013 (try again later) service_unavailable; then sent 1013 (try again later) service_unavailable 13 sh[523528] 01 22 32 080 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Exiting worker context 14 sh[523528] 01 22 32 081 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Tearing down worker 15 sh[523528] 01 22 32 083 | DEBUG | prefect runner - Stopping runner 16 sh[523528] 01 22 32 083 | DEBUG | prefect FlowRunCancellingObserver - Shutting down FlowRunCancellingObserver 17 sh[523528] 01 22 32 084 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Emitting event id=019e77fa-4993-76b9-9b76-bea0068d701a 18 sh[523528] 01 22 32 085 | DEBUG | prefect events clients - Added event id=019e77fa-4993-76b9-9b76-bea0068d701a to unconfirmed events list There are now 1 unconfirmed events 19 sh[523528] 01 22 32 085 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Emit reconnection attempt 0 20 sh[523528] 01 22 32 086 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Sending event id=019e77fa-4993-76b9-9b76-bea0068d701a 21 sh[523528] 01 22 32 087 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Got retryable error ConnectionClosedError 22 sh[523528] 01 22 32 097 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Emit reconnection attempt 1 23 sh[523528] 01 22 32 108 | DEBUG

Janet Carson

06/01/2026, 4:43 PM

Marvin

06/01/2026, 4:43 PM

Your message was too long, here's your message at the allowed limit: I'm looking at the logs for my prefect ProcessWorker work pool It's happily working along until line 13 -- exiting worker context -- and after that bit of code it never queries for flows again What causes this? My worker is on prefect 3 7 2 1 01 22 28 057 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Querying for flow runs scheduled before 2026-05-30 08 22 38 057320+00 00 2 01 22 28 189 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Discovered 0 scheduled_flow_runs 3 01 22 32 079 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Worker channel unhealthy REST fallback is active received 1013 (try again later) service_unavailable; then sent 1013 (try again later) service_unavailable 4 01 22 32 080 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Exiting worker context 5 01 22 32 081 | DEBUG | prefect workers process processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Tearing down worker 6 01 22 32 083 | DEBUG | prefect runner - Stopping runner 7 01 22 32 083 | DEBUG | prefect FlowRunCancellingObserver - Shutting down FlowRunCancellingObserver 8 01 22 32 084 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Emitting event id=019e77fa-4993-76b9-9b76-bea0068d701a 9 01 22 32 085 | DEBUG | prefect events clients - Added event id=019e77fa-4993-76b9-9b76-bea0068d701a to unconfirmed events list There are now 1 unconfirmed events 10 01 22 32 085 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Emit reconnection attempt 0 11 01 22 32 086 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Sending event id=019e77fa-4993-76b9-9b76-bea0068d701a 12 01 22 32 087 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Got retryable error ConnectionClosedError 13 01 22 32 097 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Emit reconnection attempt 1 14 01 22 32 108 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Attempting websocket reconnection 15 01 22 32 113 | DEBUG | prefect events clients - Reconnecting websocket connection 16 01 22 32 119 | DEBUG | prefect events clients - Cleared existing websocket connection 17 01 22 32 125 | DEBUG | prefect events clients - Opening websocket connection 18 01 22 32 349 | DEBUG | prefect events clients - Pinging to ensure websocket connected 19 01 22 32 450 | DEBUG | prefect events clients - Pong received Websocket connected 20 01 22 32 461 | DEBUG | prefect events clients - Resending 1 unconfirmed events 21 01 22 32 467 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Emitting event id=019e77fa-4993-76b9-9b76-bea0068d701a 22 01 22 32 472 | DEBUG | prefect events clients - Added event id=019e77fa-4993-76b9-9b76-bea0068d701a to unconfirmed events list There are now 1 unconfirmed events 23 01 22 32 477 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Emit reconnection attempt 0 24 01 22 32 483 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Sending event id=019e77fa-4993-76b9-9b76-bea0068d701a 25 01 22 32 509 | DEBUG | prefect events clients - EventsClient(id=137765344224272) Checkpointing event id=019e77fa-4993-76b9-9b76-bea0068d701a 26 01 22 32 515 | DEBUG

Janet Carson

06/01/2026, 4:46 PM

@Marvin These are the logs for a self-hosted ProcessWorker. It's working until it logs "Exiting worker context" -- and after that never queries for flows again. The process is still running in a hot loop consuming CPU. How to debug? My worker is on prefect 3 7 2 012228.057 prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Querying for flow runs scheduled before 2026-05-30 082238.057320+00000122:28.189 prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Discovered 0 scheduled_flow_runs 012232.079 prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Worker channel unhealthy, REST fallback is active: received 1013 (try again later) service_unavailable; then sent 1013 (try again later) service_unavailable012232.080 prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Exiting worker context... 012232.081 prefect.workers.process.processworker 43d1559c-f719-401e-92bc-f1bc24e95885 - Tearing down worker... 012232.083 prefect.runner - Stopping runner... 012232.083 prefect.FlowRunCancellingObserver - Shutting down FlowRunCancellingObserver 012232.084 prefect.events.clients - EventsClient(id=137765344224272): Emitting event id=019e77fa-4993-76b9-9b76-bea0068d701a. 012232.085 prefect.events.clients - Added event id=019e77fa-4993-76b9-9b76-bea0068d701a to unconfirmed events list. There are now 1 unconfirmed events. 012232.085 prefect.events.clients - EventsClient(id=137765344224272): Emit reconnection attempt 0. 012232.086 prefect.events.clients - EventsClient(id=137765344224272): Sending event id=019e77fa-4993-76b9-9b76-bea0068d701a. 012232.087 prefect.events.clients - EventsClient(id=137765344224272): Got retryable error: ConnectionClosedError 012232.097 prefect.events.clients - EventsClient(id=137765344224272): Emit reconnection attempt 1. 012232.108 prefect.events.clients - EventsClient(id=137765344224272): Attempting websocket reconnection.012232.113 prefect.events.clients - Reconnecting websocket connection. 012232.119 prefect.events.clients - Cleared existing websocket connection. 012232.125 prefect.events.clients - Opening websocket connection. 012232.349 prefect.events.clients - Pinging to ensure websocket connected. 012232.450 prefect.events.clients - Pong received. Websocket connected. 012232.461 prefect.events.clients - Resending 1 unconfirmed events. 012232.467 prefect.events.clients - EventsClient(id=137765344224272): Emitting event id=019e77fa-4993-76b9-9b76-bea0068d701a. 012232.472 prefect.events.clients - Added event id=019e77fa-4993-76b9-9b76-bea0068d701a to unconfirmed events list. There are now 1 unconfirmed events. 012232.477 prefect.events.clients - EventsClient(id=137765344224272): Emit reconnection attempt 0. 012232.483 prefect.events.clients - EventsClient(id=137765344224272): Sending event id=019e77fa-4993-76b9-9b76-bea0068d701a. 012232.509 prefect.events.clients - EventsClient(id=137765344224272): Checkpointing event id=019e77fa-4993-76b9-9b76-bea0068d701a.012232.515 prefect.events.clients - Finished resending unconfirmed events. 012232.520 prefect.events.clients - EventsClient(id=137765344224272): Sending event id=019e77fa-4993-76b9-9b76-bea0068d701a. 012232.546 prefect.events.clients - EventsClient(id=137765344224272): Checkpointing event id=019e77fa-4993-76b9-9b76-bea0068d701a. 012302.559 prefect.events.clients - Time-based checkpoint: confirming 1 events. 012302.665 prefect.events.clients - EventsClient(id=137765344224272): Pong received. Events checkpointed.

Marvin

06/01/2026, 4:56 PM

✅ thought for 192.3 seconds

Marvin

06/01/2026, 5:00 PM

What you’re seeing does not look like normal worker behavior. The important sequence is:

Copy code

text
Worker channel unhealthy, REST fallback is active: received 1013 ... service_unavailable
Exiting worker context...
Tearing down worker...
Stopping runner...
...

After that, a healthy worker should either: 1. continue polling via REST, or 2. fully exit so your supervisor restarts it Instead, yours appears to get stuck after teardown starts, while the process stays alive and spins CPU. That suggests the worker is hanging during shutdown/cleanup rather than returning to normal polling. My best read: this is likely a bug in 3.7.2 around worker-channel/websocket failure handling and teardown. A few useful points: -

1013 service_unavailable

is a websocket-side “try again later” signal - Prefect logs that REST fallback is active, but then immediately logs

Exiting worker context...

- your later logs are mostly from the events client reconnecting, not from the worker’s flow-run polling loop - since you never see

Querying for flow runs...

again, the main polling loop likely stopped I wasn’t able to confirm an exact GitHub issue because GitHub search failed from my side, but the source-code research strongly suggests this is in the worker/channel lifecycle rather than your deployment config. What I’d do first 1. Upgrade off 3.7.2 if possible - Even if there isn’t a confirmed one-line fix in 3.7.3 specifically, I would not stay on 3.7.2 for this. - If you can reproduce on the latest 3.x, that’s much more actionable. - Release notes: Prefect releases 2. Run the worker with debug logging Set:

Copy code

bash
   PREFECT_LOGGING_LEVEL=DEBUG
   PREFECT_DEBUG_MODE=1

Then capture the full sequence from startup through the “Exiting worker context...” line. 3. Get a Python stack dump from the stuck process This is the fastest way to tell what loop is spinning. Start the worker with:

Copy code

bash
   PYTHONFAULTHANDLER=1

Then when it gets stuck, send:

Copy code

bash
   kill -USR1 <pid>

or sometimes:

Copy code

bash
   kill -USR2 <pid>

depending on your environment / handler setup. Also, if you can attach with

py-spy

, that’s even better:

Copy code

bash
   py-spy dump --pid <pid>
   py-spy top --pid <pid>

4. Check whether it’s spinning in websocket/events code Your logs already suggest that at least part of the remaining activity is in

prefect.events.clients

. If

py-spy top

shows time concentrated there, that’s a strong clue. 5. Use an external supervisor Until root cause is resolved, make sure the worker is restarted if it stops making progress. - systemd

Restart=always

- Docker/Kubernetes restart policy - liveness check based on recent polling / heartbeat logs Why this looks buggy Based on code-path research, the likely failure mode is: - worker websocket becomes unhealthy - worker begins teardown / exits its context - a background task tied to the worker channel or events client keeps running/retrying - the main polling loop never resumes - process remains alive, so your service manager doesn’t necessarily restart it - CPU rises because it’s stuck in a retry/spin loop Concrete debugging checklist Please collect these from one stuck instance:

Copy code

bash
prefect version
python --version
pip show prefect
env | grep PREFECT

Then while stuck:

Copy code

bash
ps -fp <pid>
top -Hp <pid>
py-spy dump --pid <pid>
py-spy top --pid <pid>
strace -p <pid>

If you’re in containers:

Copy code

bash
kubectl logs <pod> --previous=false
kubectl describe pod <pod>

Things to look for specifically - Is there a proxy / load balancer in front of the Prefect API that may be interrupting websockets? - Does this happen only against self-hosted server, or also against Cloud? - Are there ingress timeouts, idle timeouts, or websocket upgrade restrictions?

Marvin

06/01/2026, 5:00 PM

- Do you see repeated 1013s across multiple workers, or just this one? - Does disabling event/cancellation-related functionality change the behavior? Infra suspicion: websocket path/proxy Because the trigger is

1013 service_unavailable

, I’d also inspect anything between worker and API: - nginx / ALB / ingress websocket support - idle timeout settings - reverse proxy buffering / upgrade headers - server overload around the time of failure If you’re self-hosting Prefect Server behind ingress, a websocket-unfriendly proxy could absolutely be part of this. Recommended next step If you want, send me: - how you start the worker - whether you use Prefect Cloud or self-hosted server - whether there is nginx / ALB / ingress in front of the API - a

py-spy dump

or faulthandler stack from the stuck process and I can help narrow down whether this is: 1. worker bug, 2. events-client bug, or 3. websocket/proxy infrastructure causing the bad state.

Janet Carson

06/01/2026, 5:06 PM

@Marvin locally hosted worker, prefect cloud server Worker is started with systemd, command is: prefect worker start --with-healthcheck --pool pool-name The messages about websocket unstable, falling back to REST are continuous in the logs. This error takes hours between failures - is there a faster way to get there?

Marvin

06/01/2026, 5:07 PM

✅ thought for 30.9 seconds

Marvin

06/01/2026, 5:07 PM

Yes — if the websocket instability messages are continuous, you can usually force the bad path much faster by breaking websocket connectivity instead of waiting for the organic failure. Since you’re on a locally hosted worker talking to Prefect Cloud, the fastest reproduction methods are network-level. Fastest ways to trigger it 1. Temporarily block the worker’s egress to Prefect Cloud On the worker host, briefly drop outbound traffic to the API/websocket endpoint, then restore it. For example, with

iptables

/ firewall rules, block traffic for ~30–60 seconds, then remove the rule. If you don’t want to mess with firewall rules, an easier version is: - stop outbound network from that VM/container temporarily - or kill the route to the internet briefly - or use a proxy in front of the worker and bounce the proxy 2. Blackhole DNS for the Prefect API host If the worker resolves the Cloud endpoint through local DNS, temporarily override resolution to a dead IP, then restore it. 3. Use
tc netem
to inject network faults This is often the cleanest reproducible test: - add packet loss - add high latency - add connection resets Example idea:

Copy code

bash
   tc qdisc add dev eth0 root netem loss 30% delay 1000ms 500ms
   # let it run briefly
   tc qdisc del dev eth0 root

4. Send `SIGSTOP`/`SIGCONT` to the process Less realistic, but sometimes enough to trigger heartbeat / websocket unhealthy behavior:

Copy code

bash
   kill -STOP <pid>
   sleep 60
   kill -CONT <pid>

Best reproduction approach If your goal is “get to failure fast and inspect it,” I’d do this: 1. run one dedicated worker with:

Copy code

bash
   PREFECT_LOGGING_LEVEL=DEBUG
   PREFECT_DEBUG_MODE=1
   PYTHONFAULTHANDLER=1
   prefect worker start --with-healthcheck --pool pool-name

2. induce a short network failure for 30–120s 3. when it reaches the stuck state, immediately collect:

Copy code

bash
   py-spy top --pid <pid>
   py-spy dump --pid <pid>

That should tell us whether it’s spinning in: - worker channel code - events client websocket code - healthcheck server loop - some AnyIO cancellation edge case Given your new detail, here’s my updated hypothesis Because: - worker is local - backend is Prefect Cloud - websocket unstable/fallback messages are continuous this now looks even more like a websocket/channel problem than a scheduling problem. In other words: - the worker may be surviving on REST fallback for a long time - eventually one of those repeated websocket/channel failures puts it into the bad teardown state - after that it stops polling altogether So the repeated unhealthy websocket messages are probably the precursor, not just noise. Two things I’d specifically check 1. Does the host have anything interfering with long-lived outbound websocket connections? Examples: - corporate proxy - egress filtering - TLS inspection - unstable NAT/firewall idle handling - VPN 2. Does systemd think the service is healthy after this happens? Since you’re using

--with-healthcheck

, if the process is stuck but the healthcheck still passes, that’s important. Check whether the health endpoint still returns healthy after the worker stops polling. Practical debugging with systemd When it fails, capture:

Copy code

bash
systemctl status <service-name>
journalctl -u <service-name> -n 300 --no-pager
curl <http://127.0.0.1:<healthcheck-port>/health>

If the healthcheck still says healthy while the worker is no longer polling, that’s a useful signal. If you want the quickest minimal repro The simplest likely-to-work test is:

Copy code

bash
# start worker with debug logging
PREFECT_LOGGING_LEVEL=DEBUG PREFECT_DEBUG_MODE=1 prefect worker start --with-healthcheck --pool pool-name

Then from another shell, interrupt network for ~1 minute:

Copy code

bash
kill -STOP <pid>
sleep 60
kill -CONT <pid>

Marvin

06/01/2026, 5:07 PM

If that doesn’t do it, move to actual network impairment via

tc

or firewall rules. If you want, I can give you: - a safe
tc netem
recipe to simulate Cloud websocket instability - a systemd unit override that auto-restarts on this stuck condition - or a step-by-step capture plan for the next repro with

py-spy

and healthcheck output

Open in Slack

Previous Next