A few things can cause this pattern on Windows, especially when the service process stays up but the worker itself disappears from Prefect. Most likely causes: 1. The worker stops heartbeating to Prefect If the worker’s connection to the API/WebSocket gets interrupted or hangs, the process can remain alive but Prefect will mark it offline. This can happen because of: - network devices closing long-lived connections - proxy / firewall / VPN timeouts - transient DNS or connectivity issues - the worker event loop getting stuck 2. The worker is hung, not dead On Windows services, it’s possible for the Python process to keep running while the worker loop stops polling / heartbeating. 3. Service wrapper issues If you’re using a Windows service wrapper, stdout/stderr handling, environment loading, working directory, or session differences can cause subtle hangs or broken reconnect behavior over time. What I’d recommend: 1. Turn on debug logging and capture logs to file You want to confirm whether heartbeats/polling stop, whether reconnect attempts are happening, or whether the process is just stuck. If you’re running the worker from a service manager, make sure stdout/stderr are redirected to files. Environment variables to try:

PREFECT_LOGGING_LEVEL=DEBUG
PYTHONUNBUFFERED=1

Look for messages about: - heartbeat failures - reconnects - websocket disconnects - API timeouts - worker polling stopping 2. Run the worker with a healthcheck endpoint This gives you a way to tell whether the worker loop is actually healthy, instead of only checking whether the Windows service is still running. Before suggesting the command, I’d verify the CLI syntax, but in general for Prefect 3 the worker supports

--with-healthcheck

. If you want, I can give you the exact verified command for your worker type / pool setup. Relevant docs: - Worker healthchecks - Workers overview 3. Add automatic restart if the worker becomes unhealthy Since the process may stay alive while the worker is unusable, a service manager should restart it based on health, not just process existence. On Windows, common approaches are: - NSSM - Windows Task Scheduler - another service wrapper with restart-on-failure and log capture If possible, have the service wrapper or an external watchdog restart the worker when the health endpoint fails. 4. Check for a 6-hour network timeout The “about 6 hours” detail is suspicious. That often points to infrastructure timing out long-lived connections. Things to check: - corporate proxy - firewall - VPN - load balancer - endpoint security software - idle TCP/WebSocket timeout If you’re behind any of those, the worker may lose its live connection and fail to recover cleanly. 5. Make sure the service environment is complete Windows services often run with a different environment than an interactive shell. Verify the service has: -

PREFECT_API_URL

-

PREFECT_API_KEY

if using Cloud - correct working directory - same Python environment / venv as your manual tests - any proxy env vars needed for outbound traffic 6. Compare “interactive shell” vs “service” behavior A very useful test: - run the same worker command in a normal terminal for 8+ hours - compare that with the service run If it only fails as a service, that strongly suggests a Windows service/session/wrapper/environment issue rather than Prefect itself. Best practical fix If this is production-critical, I’d recommend: - enable debug logs - enable healthchecks - use a watchdog/service wrapper that restarts on unhealthy state - investigate proxy/firewall timeouts around the 6-hour mark If you want, I can help you with either of these next: 1. a verified Prefect 3 worker command for Windows service use, or 2. a step-by-step NSSM setup with logging and restart behavior.

That’s a very useful clue. If: - NSSM stays running - worker logs show no shutdown/error - UI says worker is not ready - runs become Late then the most likely explanation is: the worker process is still alive, but it has stopped successfully heartbeating and/or polling Prefect. In Prefect, “ready” is not based on “is the process alive on Windows”; it’s based on whether Prefect is still receiving recent worker heartbeats and queue polls. If those stop, the UI marks the worker/pool not ready even if

python.exe

is still running. What this usually means - the worker event loop is hung or stalled - network/API communication is silently broken - the worker stopped polling for work - the worker stopped heartbeating - stdout logging is still fine, so you don’t necessarily see a crash A few important details: Why jobs become

Late

If the worker stops polling the work pool/queue, Prefect has nobody actively claiming scheduled runs. They stay scheduled past their expected start time and become

Late

. Why you may see no logs If the worker is stuck in a hung async state, deadlocked, or blocked on a call that doesn’t error, it may never log a failure. “No logs” is actually consistent with a hang. Most likely causes in your case Given the ~6 hour timing and Windows service setup, I’d prioritize these: 1. Network connection silently dies A firewall, proxy, VPN, load balancer, or NAT timeout may drop a long-lived connection. If reconnect behavior doesn’t recover cleanly, the process stays up but Prefect stops hearing from it. 2. Worker loop hangs The async polling / heartbeat loop may get stuck. This is especially plausible if there are environmental issues specific to Windows services. 3. Something blocks API calls indefinitely If an HTTP/WebSocket operation hangs instead of failing fast, the worker can appear frozen without logging an exception. What I’d do next 1. Check whether heartbeats/polling actually stop at a specific timestamp In the UI or API, compare: - last time worker was seen - last time work queue was polled - timestamp of the last normal worker log line If those all stop around the same moment, it strongly points to a stuck worker loop or broken connectivity. 2. Enable healthchecks and monitor them externally If not already using it, start the worker with a healthcheck endpoint so you can tell: - process alive - versus worker actually healthy If the healthcheck starts failing while NSSM still shows the service up, that confirms the worker loop is unhealthy rather than exited. Docs: - Worker healthchecks 3. Make NSSM restart based on health, not just process existence Right now NSSM is only telling you “the process still exists.” That is not enough for Prefect workers. You want either: - an external watchdog that calls the worker health endpoint and restarts the service if unhealthy - or scheduled forced restarts as a short-term mitigation 4. Check whether the issue reproduces outside NSSM Run the exact same worker command in a normal terminal for 8+ hours. If it only happens under NSSM/service context, then the problem is likely: - service environment - session differences - networking/proxy for service accounts - Windows service behavior If it also happens interactively, the issue is more likely: - Prefect worker bug - network path issue - machine-level connectivity problem 5. Force shorter network timeouts / better diagnostics This is often helpful: - debug logging - confirm no proxy for the service account unless intended - verify antivirus/security tooling is not interfering with long-lived Python connections A few concrete things to inspect Service account Is the service running as: - LocalSystem - a domain user - your user account Service accounts often have different: - proxy settings - cert stores - DNS behavior - firewall rules - network permissions

That difference alone can explain “works for hours, then goes not ready.” Any proxy/VPN/security appliance The “about 6 hours” timing is a big hint. That smells like an idle/session timeout somewhere in the network path. Temporary mitigation Until root cause is confirmed, the most practical mitigation is: - enable healthcheck - restart on failed healthcheck - or schedule a periodic worker restart every few hours That’s not ideal, but it’s often effective while debugging. If you want, I can help with one of these next: 1. a verified Prefect 3 worker command with healthcheck for Windows 2. a PowerShell watchdog script that restarts the NSSM service when the healthcheck fails 3. steps to inspect the worker last-heartbeat / last-polled timestamps to confirm exactly what’s stopping

That log is the key.

1013 service_unavailable

is a WebSocket close code meaning the server side told the worker: try again later. In practice, Prefect’s worker channel became unavailable, so the worker dropped back to REST heartbeats/polling. So this is not “the Windows service stopped.” It means the worker’s real-time channel to the Prefect API/server was closed. Why that can happen: 1. Temporary server/API unavailability If you’re using Prefect Cloud or a self-hosted API behind ingress/load balancers, the worker channel can get closed when: - the API is restarting - the server is overloaded - an upstream proxy/load balancer is refusing or recycling connections - websocket support is flaky in the network path 2. Reverse proxy / ingress doesn’t handle WebSockets well This is a very common cause for self-hosted Prefect. If you have nginx, Traefik, Azure App Gateway, ALB, Cloudflare, corporate proxy, etc., it may: - time out the websocket - not forward upgrade headers correctly - close long-lived connections - shed connections under load 3. Prefect server is reachable, but the worker channel endpoint is not healthy The worker may still be able to do some normal HTTP traffic, but the websocket channel specifically is degraded. 4. Resource pressure on the API/server side If the server can’t keep up, it may reject or close worker channel connections with “try again later”. Why this matters - The worker channel is used for real-time coordination - when it becomes unhealthy, the worker switches to REST fallback - if REST fallback also struggles or hangs, then the worker eventually appears offline / not ready and runs go Late So this log is strong evidence that the problem is connectivity or server-side channel availability, not just NSSM. What to check next depends on whether you use Prefect Cloud or self-hosted Prefect Server. If you use Prefect Cloud Most likely causes are: - outbound proxy/firewall/VPN interference - intermittent network path issues from that Windows host - TLS inspection/security appliance interfering with websockets Check: - whether the machine is behind a corporate proxy - whether the service account has different proxy/cert behavior than your interactive user - whether the issue lines up with VPN reconnects or security policies - whether general outbound connectivity drops around the same time If you use self-hosted Prefect Server Check your ingress / proxy immediately. Common problems: - websocket upgrades not configured correctly - read/send timeout too short - idle timeout around hours - pod/server restarts - worker channel endpoint overloaded You’d want to inspect: - Prefect server logs - reverse proxy logs - ingress timeout settings - restarts / OOMs / scaling events around the time of the

1013

What the code means operationally That message:

text
Worker channel unhealthy, REST fallback is active: received 1013 (try again later) service_unavailable; then sent 1013 (try again later) service_unavailable

basically means: 1. worker had a websocket/channel connection 2. server or upstream closed it with

1013

3. worker recognized the channel as unhealthy 4. worker switched to REST fallback mode That should allow continued operation, but if fallback doesn’t keep working reliably, the worker later becomes not ready. Best next steps 1. Determine Cloud vs self-hosted That changes the likely root cause a lot. 2. Correlate timestamps At the exact time of the

1013

, check: - Prefect API/server logs - proxy/ingress logs - Windows event logs - any VPN/proxy/security tool events 3. Test outside the service account Run the worker interactively on the same machine and see whether the same

1013

appears after similar uptime. 4. Add healthcheck + auto-restart Even if we find the root cause, this is still a good safety net. 5. If self-hosted, inspect websocket support in front of Prefect This is the first place I’d look.

If you tell me whether you’re using Prefect Cloud or self-hosted server, I can give you a much more targeted checklist.

Got it — on Prefect Cloud, that

1013 service_unavailable

usually points to a problem on the network path from your Windows host to Cloud, not NSSM itself. Most likely causes on Cloud: 1. Corporate proxy / TLS inspection / firewall This is the top suspect, especially on Windows service accounts. A service may use different: - proxy settings - certificate trust store behavior - outbound rules - identity/VPN policy So the worker can run fine for a while, then its websocket gets dropped or degraded. 2. VPN / security agent interference If the machine is on VPN, or has endpoint security software, long-lived websocket connections may be reset periodically. 3. Local networking issue on that host DNS hiccups, NIC resets, sleep/power policies, or ephemeral connection issues can break the worker channel. What the

1013

means in Cloud It means the worker channel became unavailable and the worker switched to REST fallback. That is usually recoverable, but if the machine/network keeps interfering, the worker may eventually stop heartbeating/polling enough to be marked not ready. What I’d check first 1. Is the service running under a different account than your interactive shell? If yes, compare them carefully. The service account may have different: - proxy config - VPN access - Windows cert store access - firewall policy This is very often the reason service behavior differs from terminal behavior. 2. Is there any outbound proxy configured? Check environment variables and Windows proxy settings for the service account: -

HTTP_PROXY

-

HTTPS_PROXY

-

NO_PROXY

Also check machine-level WinHTTP proxy:

powershell
netsh winhttp show proxy

If there is TLS inspection or an authenticating proxy, that can absolutely cause websocket instability. 3. Does it happen if you run the worker interactively on the same machine for 8+ hours? This is the cleanest isolation test. - If interactive is stable, the issue is probably service-account/proxy/environment related. - If interactive also fails, the issue is probably machine/network/security-path related. 4. Check whether the machine sleeps or changes network state Even a “server-like” Windows box can have: - NIC power saving - VPN refreshes - network profile changes - GPO/security refresh behavior 5. Add a watchdog restart Short-term mitigation: - enable worker healthcheck - monitor it externally - restart the service when unhealthy That won’t solve the network issue, but it will reduce Late runs. What I’d recommend as the likely fix path Best next diagnostic steps: 1. run the worker interactively on the same host 2. compare service account vs your user account 3. inspect proxy/VPN/security tooling 4. add healthcheck-based restart If you want, I can help you with either: - a PowerShell checklist to compare network/proxy settings for the service account - or a watchdog script to restart the worker service when healthchecks fail Useful docs: - Worker healthchecks - Prefect workers

That changes the picture quite a bit. If it also fails when started from a normal terminal, then this is probably not NSSM-specific. And since it started after a Python upgrade, the Python version is now a top suspect. Most likely explanation: - the upgraded Python version changed async/network behavior on Windows - after hours of runtime, the worker’s connection or event loop degrades - Prefect Cloud closes the worker channel with

1013 service_unavailable

- REST fallback eventually isn’t enough, so the worker becomes not ready and runs go

Late

So the two strongest hypotheses now are: 1. Python version compatibility/regression on Windows This is especially plausible if you moved: - from 3.10 -> 3.11 - 3.11 -> 3.12 - or to 3.13 very early in its lifecycle Windows asyncio behavior has changed across Python releases, and long-running websocket/async workloads are exactly where regressions tend to show up. 2. Dependency mismatch after Python upgrade If you upgraded Python and reinstalled into a fresh environment, you may also have changed: -

prefect

-

anyio

-

httpx

-

websockets

-

uvicorn

or other transitive deps So the culprit may be the new Python + new dependency set rather than Python alone. What I’d do next, in order: 1. Tell me exactly which Python version you upgraded from and to That’s the single most important data point now. Example: -

3.10.11 -> 3.12.4

-

3.11.x -> 3.12.x

2. Check your Prefect version too Run:

prefect version
python --version

If you paste both, I can give more targeted advice. 3. Most important test: try the previous Python version again If possible, recreate the worker environment on the old Python version and run the same worker command. If the problem disappears, you’ve basically isolated it. 4. Pin Prefect and dependencies in a clean venv After a Python upgrade, it’s worth rebuilding the environment cleanly instead of reusing an older one. 5. If you upgraded to Python 3.12+, consider testing 3.11 or 3.10 As a practical workaround, I would strongly consider running the worker on the previous stable Python version that you know worked, at least until the root cause is confirmed. Short version: If this started immediately after a Python upgrade and now reproduces both in NSSM and in a normal terminal, I would treat the Python/runtime change as the likely trigger. Please send: - output of

python --version

- output of

prefect version

- what version you were on before and I’ll help narrow down whether this looks like: - an unsupported Python version - a Windows asyncio regression - or a dependency version issue.

Thanks — that’s very helpful. The most likely issue is: Python 3.12 on Windows. Your setup: -

Prefect 3.7.1

-

Python 3.12.5

-

Prefect Cloud

And your symptoms: - long-running worker goes unhealthy after many hours -

1013 service_unavailable

- reproduces even outside NSSM That combination points much more toward a runtime-level Windows async/network issue than a service-wrapper issue. What I think is happening - Prefect workers are long-running async processes - on Windows, they rely on Python’s asyncio event loop behavior - Python 3.12 changed a number of asyncio internals - after hours, the worker channel/websocket path appears to degrade or stall - Cloud closes the worker channel, the worker falls back to REST, and eventually the worker is treated as not ready Because it happens in a normal terminal too, this is almost certainly not just NSSM. Most practical recommendation: try Python 3.11 If this were my system, the first thing I’d do is test the exact same worker on Python 3.11 in a clean virtual environment. I would treat that as the best next step, not as a last resort. Why 3.11? - still modern and widely used - generally safer on Windows for long-lived async workloads than 3.12 when odd regressions appear - easiest way to confirm whether the Python upgrade introduced the issue What I’d do exactly 1. Create a clean 3.11 environment Install Python 3.11, then create a fresh venv and reinstall Prefect. Example:

py -3.11 -m venv .venv311
.venv311\Scripts\activate
python -m pip install -U pip
pip install "prefect==3.7.1"
prefect version

Then start the worker with the same configuration and let it run past the failure window. 2. Don’t reuse the 3.12 environment Use a fresh environment so you don’t carry over dependency oddities. 3. If 3.11 is stable, keep workers on 3.11 for now That would strongly suggest a 3.12-specific runtime interaction. 4. Also consider upgrading Prefect after testing You’re on

3.7.1

. If there’s a newer 3.x release available, it may include worker-channel fixes or dependency updates. I can check the latest release for you if you want. A couple more notes The

Server: Database sqlite

line is not the Cloud database That section is about local/server-related package info in the CLI output, not saying your Cloud backend is SQLite. So that part is not the problem here. This doesn’t look like misconfiguration Because: - it worked before - it fails in service and interactive modes - it started after the Python upgrade - the log shows worker channel degradation rather than auth/config errors My recommendation in one line: Move the worker back to Python 3.11 and retest before spending more time debugging NSSM or Prefect Cloud networking. If you want, I can help with either: 1. a clean Python 3.11 reinstall plan for the worker, or 2. checking whether there’s a newer Prefect release worth testing on top of that.

If Python 3.12 + Prefect works on one machine but not your local Windows PC, then the issue is probably not “Prefect doesn’t support 3.12” in a simple global sense. It usually means the Python upgrade exposed a machine-specific difference on your local PC. Most likely causes: 1. Different dependency versions after the upgrade Even if both say “Prefect 3.7.1”, the full environment may differ. One PC might have different versions of: -

anyio

-

httpx

-

httpcore

-

websockets

-

sniffio

-

certifi

-

uvicorn

That can absolutely change long-running websocket behavior. This is one of the most common causes. Check with:

pip freeze

Compare the working server vs failing local PC. 2. Different network path Your local PC may be subject to things the server is not: - corporate proxy - VPN - endpoint security / antivirus - TLS inspection - desktop firewall rules - Wi-Fi instability - power management on the NIC A server usually has a cleaner, more stable outbound network path than a desktop/laptop. This fits your

1013 service_unavailable

symptom very well. 3. Different Windows environment Your local PC may differ in: - Windows version/build - patches - certificate store - proxy configuration - DNS resolver behavior - network adapter driver - sleep/power settings So Python 3.12 may only surface a latent issue on that machine. 4. Different Python build/install Not all Python installs are effectively identical. Differences can include: - installed from python.org vs Microsoft Store - 32-bit vs 64-bit - per-user vs system install - different OpenSSL bundle / cert handling - old files left behind from a previous install 5. Local environment contamination If your local machine has had multiple Python installs/upgrades over time, you may have: - stale packages - multiple

python

/

pip

paths - mixed user-site and venv packages - leftover wheels compiled against different versions This often appears right after “just a Python upgrade.” 6. Security software hooks into Python/network traffic On desktops especially, antivirus/EDR products can interfere with long-lived Python network connections more than on servers. Why it was working before the Python version change Because the Python upgrade may not have created the underlying machine difference — it may have simply made it visible. Examples: - Python 3.11 tolerated a flaky network condition better - Python 3.12 changed timing/reconnect behavior enough to trigger the issue - a dependency got upgraded along with Python and now behaves differently - the new Python install uses a different SSL/cert/proxy path So “worked before upgrading Python” is fully consistent with a local-machine-specific cause. What I would compare first On both machines, compare these exactly: Python and Prefect

python --version
prefect version

All installed package versions

pip freeze

Proxy-related environment

set HTTP_PROXY
set HTTPS_PROXY
set NO_PROXY

WinHTTP proxy

netsh winhttp show proxy

Python executable path

where python
python -c "import sys; print(sys.executable)"

OpenSSL version used by Python

python -c "import ssl; print(ssl.OPENSSL_VERSION)"

That comparison will often reveal the difference. Most likely root cause ranking from what you’ve shared: 1. different dependency versions 2. local PC network/proxy/security software 3. dirty/local Python environment after upgrade 4. Windows-specific local machine issue Best next step The highest-value step is to compare: -

pip freeze

on working server -

pip freeze

on failing local PC If you want, paste both and I can help identify the important diffs. Also useful would be: - whether your local PC is on Wi-Fi or VPN - whether the server is Windows too - whether the server uses the same Python 3.12.x and same Prefect version

If the environment is now clean and the worker initially connects but later stops polling, then you’re back to the earlier class of issue: the worker process is alive, but the polling / heartbeat loop becomes unhealthy over time. Given everything you’ve shared, the most likely causes are now: 1. Python 3.12 on your local Windows machine is still the trigger Even with the environment fixed, if this only started after the Python upgrade and the worker eventually stops polling, I would still treat

Python 3.12 + this Windows host

as the primary suspect. 2. Local machine/network interference Since you’re on Prefect Cloud, the worker depends on stable long-lived outbound connectivity. If your local PC has: - VPN - proxy - endpoint protection - aggressive firewalling - Wi-Fi/power management it can interrupt the worker channel and eventually polling too. 3. Worker channel degrades, REST fallback doesn’t sustain execution indefinitely You already saw:

Worker channel unhealthy, REST fallback is active: received 1013 ... service_unavailable

That means the real-time channel is breaking. If the machine/network keeps causing trouble, the worker can eventually stop polling effectively even though it started fine. At this point, the two highest-value actions are: 1. Try the same worker on Python 3.11 on this PC This is still the cleanest experiment. If

3.11

runs stably and

3.12.5

does not, you’ve isolated the trigger enough to use 3.11 as the workaround. 2. Add worker healthchecks and auto-restart Even if we don’t yet know whether it’s Python, network, or both, this gives you operational stability. Short-term answer: - if the worker stops polling after hours, restart it automatically when unhealthy Useful docs: - Worker healthchecks What I’d specifically test next Test A — same PC, Python 3.11 Fresh venv, same Prefect version, same worker command. Test B — same PC, wired network / no VPN if possible If you’re on Wi-Fi or VPN, that’s worth eliminating. Test C — check whether health endpoint fails before UI says not ready That tells you whether the worker loop itself is hanging. Test D — scheduled restart as a temporary mitigation If failure happens after many hours, a periodic restart can keep work flowing while you narrow root cause. My current best guess Based on the full thread, my ranking is: 1. Python 3.12-specific runtime behavior on your local Windows machine 2. network/security software on that machine interfering with long-lived Cloud connections 3. less likely: Prefect config issue If you want, I can give you either: - a clean Python 3.11 test setup for the worker, or - a Windows watchdog / restart script for a worker that stops polling.

That strongly suggests the worker is hanging silently, not exiting cleanly and not throwing a logged exception. If the last logs are just:

Discovered type 'process' for work pool 'devpool'.
Worker 'ProcessWorker ...' started!

and then later the UI says Not Ready, that usually means: - the worker started successfully - then one of its background loops stopped making progress - no exception got surfaced to logging - Prefect stopped receiving heartbeats / polls - the UI marked it not ready based on stale heartbeat/poll timestamps So the important point is: “Not Ready before any logged error” is exactly what you’d expect from an async loop stall or blocked network call. Why this happens A Prefect worker isn’t just one linear process; it runs background loops for things like: - polling for scheduled flow runs - heartbeating to Prefect Cloud - maintaining worker-channel connectivity If one of those loops gets stuck, you can end up with: - process still alive - no new logs - no explicit crash - UI eventually marking the worker offline/not ready That makes these the most likely causes now: 1. Event loop stall / hang on your Windows machine Still very plausible given: - the issue began after changing Python - it reproduces outside NSSM - no error is logged before readiness is lost 2. A network call is hanging rather than failing If an outbound request or websocket operation blocks indefinitely, the worker may stop progressing without logging an exception. 3. Local machine-specific interference Security software, VPN, proxy, Wi-Fi/power management, or local networking can trigger a hang pattern like this. What this tells us diagnostically This makes a pure deployment/schema/config issue much less likely. If config were wrong, you’d usually see: - auth errors - API errors - immediate failure - repeated logged retries Instead, you’re seeing: - normal startup - then silence - then stale worker state in the UI That points much more toward runtime hang than bad configuration. What I recommend now 1. Treat Python 3.12 on this machine as suspect until disproven At this point, I really would test with Python 3.11 on the same PC in a clean venv. Because: - same PC - same network - same account - same Prefect config If 3.11 works and 3.12 hangs, you have a very strong isolation result. 2. Enable healthchecks and monitor them externally This is the best way to prove “process alive but worker unhealthy.” If the health endpoint fails while the process remains running, that confirms the worker loop is stalled. 3. Add your own periodic liveness logging Since the built-in logs aren’t showing anything after startup, you may need an external signal: - monitor the health endpoint - or periodically inspect whether the process is still making network calls / CPU activity 4. Rule out local network/power interference Especially if this is a laptop or desktop: - disable sleep - disable NIC power saving - test on wired network if possible - test without VPN if possible My honest read Based on all the evidence, this does not look like a simple Prefect configuration problem. It looks like: - worker loop stall on your local Windows machine - likely triggered by Python/runtime/environment differences after the Python upgrade - with Prefect Cloud only discovering it later when heartbeats/polls stop So the best next step is still: test the same worker on Python 3.11 on this exact machine. If you want, I can help you do one of these right now: 1. set up a clean Python 3.11 worker env on Windows 2. add a healthcheck-based watchdog 3. create a tight diagnostic checklist to compare 3.11 vs 3.12 on the same PC

Those logs are actually very revealing. The key line is the first one:

Exiting worker context...

followed by:

Tearing down worker...
Stopping runner...
Shutting down FlowRunCancellingObserver
...

That is not a “worker silently stopped polling” log. That is a graceful shutdown sequence. So if those are truly the last DEBUG logs for ~10 hours, then the worker didn’t just stall there — it entered shutdown at

13:04:13

, tore itself down, and then apparently the process remained around afterward or your service wrapper left something alive. That means the real question changes to: What caused the worker to begin shutting down at 13:04? Possible causes 1. The worker received a stop/cancel signal This is the most likely interpretation of:

Exiting worker context...
Tearing down worker...

That usually happens when: - the process receives termination/shutdown - the worker main loop exits - the context manager around the worker finishes cleanly - the service wrapper or parent process triggers stop/restart behavior 2. An unhandled exception caused the worker context to unwind Sometimes the actual triggering exception happens just before the teardown sequence, and if your log capture is incomplete or buffered, you may only notice the cleanup logs. So I would look immediately before

Exiting worker context...

in the logs. 3. Another process or wrapper is signaling the worker Even if NSSM is no longer the main problem, something external could still be: - restarting the command - sending termination - replacing the process - closing its console/session - triggering a stop on idle/logout/shutdown conditions 4. Worker code intentionally exited Less common, but possible if the worker hit a code path that ended its run loop normally rather than crashing. Why this doesn’t look like the earlier “silent hang” Earlier you described: - worker stays up - UI goes not ready - no errors logged But these logs are different. These are explicit shutdown/teardown logs. So either: - this is a different failure mode than before - or the worker is exiting cleanly and the wrapper/process monitoring is masking that fact What I’d check next 1. Search the logs immediately before

Exiting worker context...

You want the preceding 50-200 lines, not the lines after it. Specifically look for: - traceback -

KeyboardInterrupt

- cancellation - connection loss - worker channel unhealthy - API errors - any line mentioning shutdown, signal, or cancellation scope The teardown lines are the effect, not the cause. 2. Confirm whether the worker process actually exited at that time On Windows, it’s possible for: - the actual worker subprocess to exit - but some parent/service process to remain alive So the UI says not ready because the worker really did stop, even if “something Prefect-ish” is still running. 3. If using NSSM, inspect NSSM logs / Windows Event Viewer around 13:04 You want to know whether NSSM: - restarted the app - saw it exit - sent a stop - encountered stdout/stderr issues - rotated or reattached process state 4. Verify whether you’re running the worker directly or through a wrapper script If a batch file / PowerShell script / launcher is involved, it may be exiting and taking the worker down cleanly. My best read from those logs Those lines suggest: the worker was told to shut down or otherwise exited its main context gracefully at 13:04. That is different from: - deadlock - poll loop stalling silently - websocket hanging without exit So the next task is not “why did polling stop?” but: what caused the worker context to exit? Most useful thing you can send next: - the 100 or so log lines immediately before

Exiting worker context...

Because the root cause is probably there. Also, if you want, I can help you interpret: - NSSM service logs - Windows Event Viewer entries - or the worker launch command/script you’re using

Right now, based on those lines alone, I would not call this a silent stall — I would call it a graceful worker shutdown.

This is much clearer now. What you’re seeing is most consistent with: the worker channel breaks, the worker falls back to REST, it keeps polling for a while, then eventually the worker shuts down cleanly. So there are two phases: Phase 1: connectivity degrades You have:

Worker channel unhealthy, REST fallback is active: received 1013 ... service_unavailable

That means the real-time worker channel to Prefect Cloud is being closed. Then you also have:

Retryable error with 1/10 attempts

So the worker is already in retry/recovery behavior. Phase 2: it keeps working in fallback mode You then see:

Discovered 0 scheduled_flow_runs
Starting run of 'get_and_submit_flow_runs'
Querying for flow runs scheduled before ...

That means the worker is still alive and polling via REST at that point. So at that moment it has not fully died yet. Phase 3: later it exits worker context And eventually you get:

Exiting worker context...
Tearing down worker...

That means the worker later left its main run context entirely. So the sequence is: 1. worker is healthy 2. websocket/channel gets closed by Cloud or network path with

1013

3. worker falls back to REST 4. worker keeps polling for some period 5. eventually the worker stops entirely / exits cleanly What likely causes that Most likely root cause: unstable connectivity to Prefect Cloud from this machine The

1013 service_unavailable

is still the anchor clue. On Prefect Cloud, that usually means one of: - local network path instability - proxy / VPN / firewall / TLS inspection - machine-specific connectivity issues - less likely, Cloud-side transient unavailability, but the fact it’s persistent on one machine makes local path more likely Because it continues polling afterward, the initial

1013

is not immediately fatal. But it suggests the worker has entered a degraded mode. Then later, something else causes the worker main context to unwind. About this line:

Unable to read server default result storage; falling back to local defaults.
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' ...

That’s also important. A

404

against the Cloud API path is not normal if your environment is fully correct and stable. That suggests one of these: - malformed or partially incorrect API URL/workspace path - stale credentials/workspace context - a request hitting the wrong endpoint - intermittent bad state in the client/session/environment Even if the worker mostly works, this kind of 404 can indicate that some requests are being formed with bad account/workspace context. That could absolutely contribute to later instability. What I think is happening overall Most likely: - the worker is running on a machine with unstable client/API connectivity - sometimes the websocket worker channel gets closed (

1013

) - the worker falls back to REST and continues polling - at some point an API request or internal service loop fails hard enough that the worker context exits - the shutdown is graceful, which is why you see teardown logs instead of a crash Why the UI says not ready before obvious errors Because once heartbeats/polling become stale, the UI marks the worker/pool not ready before the final shutdown logs show up. That part is expected. What I’d investigate next 1. The

404 Not Found

is a big deal — inspect it carefully That is the most suspicious non-websocket error you’ve shown. Please capture the full error around:

Unable to read server default result storage; falling back to local defaults.
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found'

I’d want to know: - the exact endpoint path, redacted if needed - whether it happens once at startup or repeatedly - whether it corresponds to wrong workspace/account config Because a healthy Cloud worker should not normally be hitting a bogus account/workspace path.

2. Verify the active profile and API settings in the exact shell/service Run:

prefect config view
python -c "from prefect.settings import PREFECT_API_URL; print(PREFECT_API_URL.value())"

Make sure the worker is using the exact expected workspace URL. 3. Watch for the first exception after REST fallback begins You already know the worker can survive

1013

for a while. So the real “fatal” event is probably the next error that occurs before

Exiting worker context...

. 4. Compare this machine with the working server Especially: -

prefect config view

-

pip freeze

- proxy settings - VPN/security software - network type (Wi-Fi vs wired) My current ranking of causes 1. local machine network/proxy/security interference with Prefect Cloud 2. bad or inconsistent Cloud API/workspace configuration on this machine 3. runtime issue on this Windows machine that makes recovery from degraded connectivity fail Most actionable next step The 404 is the most concrete thing to chase now. If you paste: -

prefect config view

output with secrets redacted - and the full

404

endpoint path redacted I can help determine whether there’s still a config/workspace mismatch involved, in addition to the websocket issue.