https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> Is it possible to gracefully shutdown...
# ask-marvin
t
@Marvin Is it possible to gracefully shutdown a process worker? I did nohup prefect worker start... and now I want to shut it down.
m
✅ thought for 148.8 seconds
Yes. In Prefect 3.x, process workers handle SIGTERM/SIGINT and shut down gracefully: they stop polling for new flow runs and exit once their housekeeping finishes. There isn’t a 
prefect worker stop
command, so you stop the OS process. Practical steps for a worker started with nohup: 1) Optionally pause the pool/queue first so the worker won’t pick up new runs:
Copy code
prefect work-pool pause "<your-pool>"
# or, if you target a specific queue:
prefect work-queue pause "<your-queue>" --pool "<your-pool>"
2) Find the worker PID and send SIGTERM (graceful):
Copy code
pgrep -f "prefect worker start"
kill <PID>              # default is SIGTERM (graceful)
# or
pkill -TERM -f "prefect worker start"
3) If it won’t exit after a short while, you can escalate (not recommended unless necessary):
Copy code
kill -9 <PID>           # force kill (non-graceful)
Notes: - Running flow runs launched by a process worker are subprocesses; they typically continue and report state independently, even if the worker exits. Pausing the pool/queue + SIGTERM is the cleanest approach to avoid picking up new work while shutting down. - There’s no remote “stop worker” action in the CLI or UI; use OS signals or pause the pool/queue. CLI references: - Work pool CLI (pause/resume) - Work queue CLI (pause/resume) Implementation reference: - Worker signal handling in source
t
@Marvin
Copy code
Received SIGTERM. Sending SIGINT to the None worker (PID 32068)...
Received SIGINT. Sending SIGINT to the None worker (PID 32068)...
00:16:16.976 | ERROR   | prefect.FlowRunCancellingObserver - Error consuming events
Traceback (most recent call last):
  File "/opt/conda/envs/prefect/lib/python3.12/contextlib.py", line 737, in __aexit__
    cb_suppress = await cb(*exc_details)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 776, in __aexit__
    raise exc_val
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/workers/process.py", line 162, in start
    async with anyio.create_task_group() as loops_task_group:
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 776, in __aexit__
    raise exc_val
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 744, in __aexit__
    await self._on_completed_fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/runner/_observers.py", line 56, in __aexit__
    await self._consumer_task
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/runner/_observers.py", line 27, in _consume_events
    async for event in self._events_subscriber:
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/events/clients.py", line 634, in __anext__
    await self._reconnect()
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/events/clients.py", line 566, in _reconnect
    self._websocket = await self._connect.__aenter__()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/websockets/asyncio/client.py", line 587, in __aenter__
    return await self
           ^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/_internal/websockets.py", line 135, in _proxy_connect
    proto = await self.__await_impl__()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/websockets/asyncio/client.py", line 541, in __await_impl__
    self.connection = await self.create_connection()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/websockets/asyncio/client.py", line 467, in create_connection
    _, connection = await loop.create_connection(factory, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/asyncio/base_events.py", line 1132, in create_connection
    raise exceptions[0]
  File "/opt/conda/envs/prefect/lib/python3.12/asyncio/base_events.py", line 1107, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/asyncio/base_events.py", line 1010, in _connect_sock
    await self.sock_connect(sock, address)
  File "/opt/conda/envs/prefect/lib/python3.12/asyncio/selector_events.py", line 651, in sock_connect
    return await fut
           ^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/asyncio/selector_events.py", line 691, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('10.1.37.163', 4200)
Worker 'tomworld-adventureland-2' stopped!
00:16:17.255 | ERROR   | asyncio - Task exception was never retrieved
future: <Task finished name='Task-9' coro=<critical_service_loop() done, defined at /opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/utilities/services.py:21> exception=ConnectionRefusedError(111, "Connect call failed ('10.1.37.163', 4200)")>
Traceback (most recent call last):
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/utilities/services.py", line 64, in critical_service_loop
    await workload()
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/runner/runner.py", line 1034, in _emit_flow_run_heartbeats
    await asyncio.gather(*coros)
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/runner/runner.py", line 1061, in _emit_flow_run_heartbeat
    await self._events_client.emit(
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/events/clients.py", line 166, in emit
    return await self._emit(event)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/events/clients.py", line 385, in _emit
    await self._reconnect()
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/events/clients.py", line 311, in _reconnect
    self._websocket = await self._connect.__aenter__()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/websockets/asyncio/client.py", line 587, in __aenter__
    return await self
           ^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/prefect/_internal/websockets.py", line 135, in _proxy_connect
    proto = await self.__await_impl__()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/websockets/asyncio/client.py", line 541, in __await_impl__
    self.connection = await self.create_connection()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/site-packages/websockets/asyncio/client.py", line 467, in create_connection
    _, connection = await loop.create_connection(factory, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/asyncio/base_events.py", line 1132, in create_connection
    raise exceptions[0]
  File "/opt/conda/envs/prefect/lib/python3.12/asyncio/base_events.py", line 1107, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/asyncio/base_events.py", line 1010, in _connect_sock
    await self.sock_connect(sock, address)
  File "/opt/conda/envs/prefect/lib/python3.12/asyncio/selector_events.py", line 651, in sock_connect
    return await fut
           ^^^^^^^^^
  File "/opt/conda/envs/prefect/lib/python3.12/asyncio/selector_events.py", line 691, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('10.1.37.163', 4200)
(END)
Is this something prefect did? sending sig int instead of sigterm?
m
Your message was too long, here's your message at the allowed limit: ```Received SIGTERM Sending SIGINT to the None worker (PID 32068) Received SIGINT Sending SIGINT to the None worker (PID 32068) 00 16 16 976 | ERROR | prefect FlowRunCancellingObserver - Error consuming events Traceback (most recent call last) File /opt/conda/envs/prefect/lib/python3 12/contextlib py line 737 in aexit cb_suppress = await cb(*exc_details) ^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/site-packages/anyio/_backends/_asyncio py line 776 in aexit raise exc_val File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/workers/process py line 162 in start async with anyio create_task_group() as loops_task_group ^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/site-packages/anyio/_backends/_asyncio py line 776 in aexit raise exc_val File /opt/conda/envs/prefect/lib/python3 12/site-packages/anyio/_backends/_asyncio py line 744 in aexit await self _on_completed_fut asyncio exceptions CancelledError During handling of the above exception another exception occurred Traceback (most recent call last) File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/runner/_observers py line 56 in aexit await self _consumer_task File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/runner/_observers py line 27 in _consume_events async for event in self _events_subscriber File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/events/clients py line 634 in anext await self _reconnect() File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/events/clients py line 566 in _reconnect self _websocket = await self _connect __aenter__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/site-packages/websockets/asyncio/client py line 587 in aenter return await self ^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/_internal/websockets py line 135 in _proxy_connect proto = await self __await_impl__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/site-packages/websockets/asyncio/client py line 541 in __await_impl__ self connection = await self create_connection() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/site-packages/websockets/asyncio/client py line 467 in create_connection _ connection = await loop create_connection(factory *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/asyncio/base_events py line 1132 in create_connection raise exceptions[0] File /opt/conda/envs/prefect/lib/python3 12/asyncio/base_events py line 1107 in create_connection sock = await self _connect_sock( ^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/asyncio/base_events py line 1010 in _connect_sock await self sock_connect(sock address) File /opt/conda/envs/prefect/lib/python3 12/asyncio/selector_events py line 651 in sock_connect return await fut ^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/asyncio/selector_events py line 691 in _sock_connect_cb raise OSError(err f'Connect call failed {address}') ConnectionRefusedError [Errno 111] Connect call failed ('10 1 37 163' 4200) Worker 'tomworld-adventureland-2' stopped! 00 16 17 255 | ERROR | asyncio - Task exception was never retrieved future <Task finished name='Task-9' coro=<critical_service_loop() done defined at /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/utilities/services py 21> exception=ConnectionRefusedError(111 Connect call failed ('10 1 37 163' 4200) )> Traceback (most recent call last) File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/utilities/services py line 64 in critical_service_loop await workload() File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/runner/runner py line 1034 in _emit_flow_run_heartbeats await asyncio gather(*coros) File /opt/conda/envs/prefect/lib/python3
12/site-packages/prefect/runner/runner py line 1061 in _emit_flow_run_heartbeat await self _events_client emit( File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/events/clients py line 166 in emit return await self _emit(event) ^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/events/clients py line 385 in _emit await self _reconnect() File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/events/clients py line 311 in _reconnect self _websocket = await self _connect __aenter__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/site-packages/websockets/asyncio/client py line 587 in aenter return await self ^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/site-packages/prefect/_internal/websockets py line 135 in _proxy_connect proto = await self __await_impl__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/site-packages/websockets/asyncio/client py line 541 in __await_impl__ self connection = await self create_connection() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/site-packages/websockets/asyncio/client py line 467 in create_connection _ connection = await loop create_connection(factory *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/asyncio/base_events py line 1132 in create_connection raise exceptions[0] File /opt/conda/envs/prefect/lib/python3 12/asyncio/base_events py line 1107 in create_connection sock = await self _connect_sock( ^^^^^^^^^^^^^^^^^^^^^^^^^ File /opt/conda/envs/prefect/lib/python3 12/asyncio/base_events py line 1010
t
@Marvin when I do shutdown, I got this error message: 
"/opt/conda/envs/prefect/lib/python3.12/asyncio/selector_events.py", line 691, in _sock_connect_cb raise OSError(err, f'Connect call failed {address}') ConnectionRefusedError: [Errno 111] Connect call failed ('10.1.37.163', 4200)
, that's where my prefect server is. But 
❯ nc -vz 10.1.37.163 4200 Connection to 10.1.37.163 4200 port [tcp/*] succeeded!
So my process is able to connect to it. Why is this happening?
m
✅ thought for 138.8 seconds
Short answer: this usually happens because, during shutdown, the worker tries to make a last API call (e.g., to stop polling / send final heartbeats) while the event loop is tearing down or the API is briefly unreachable. A TCP port scan with nc only proves the port is open; it doesn’t confirm the HTTP API is reachable from the worker’s network namespace or that the URL is correct. A few things to check: - Verify your API URL includes /api - In Prefect 3.x, 
PREFECT_API_URL
must include the /api suffix. For example:
Copy code
export PREFECT_API_URL=<http://10.1.37.163:4200/api>
- Inspect your current setting:
Copy code
prefect config view | grep PREFECT_API_URL
Docs: Manage settings and API health: /api/health - Test the API from the same environment as the worker - If your worker runs in a container or different VM, test from there (not just your local shell):
Copy code
curl -v <http://10.1.37.163:4200/api/health>
You should get HTTP/1.1 200 OK. If that fails but nc succeeds, it’s likely a path, proxy, or networking namespace issue. - Consider timing during shutdown - If you’re shutting down the host/VM or rapidly terminating the process, it’s normal to see a transient ConnectionRefusedError as the worker attempts a final API call during event loop teardown. - Pausing the pool/queue first can reduce activity at shutdown:
Copy code
prefect work-pool pause "<your-pool>"
  # or
  prefect work-queue pause "<your-queue>" --pool "<your-pool>"
- Double-check for proxies or firewalls - If `HTTP_PROXY`/`HTTPS_PROXY` vars are set in the worker’s environment, try unsetting them to see if it changes behavior. - Confirm any security groups/firewalls allow traffic from the worker to 10.1.37.163:4200. If the error only appears once while the worker is exiting, it’s generally harmless. If you’re seeing repeated connection errors during normal operation, please share: - The exact 
PREFECT_API_URL
value (redact credentials if any) - Where the worker runs (host/container) vs. where the server runs - The full stack trace and your Prefect version (
prefect version
) - The result of 
curl -v <http://10.1.37.163:4200/api/health>
from the worker’s environment Happy to help dig deeper with that info.
48 Views
Open in Slack
PreviousNext
Powered by