When I run with DaskTaskRunner, I seem to have som...
# ask-community
j
When I run with DaskTaskRunner, I seem to have some exception on shutdown about half the time. It doesn't seem to bother the flow, but it looks a bit alarming.
The trace looks like this:
Copy code
13:51:00.508 | INFO    | distributed.scheduler - Remove worker addr: <tcp://127.0.0.1:49720> name: 8 (stimulus_id='handle-worker-cleanup-1738090260.508733')
13:51:00.508 | INFO    | distributed.core - Connection to <tcp://127.0.0.1:49695> has been closed.
13:51:00.508 | INFO    | distributed.core - Connection to <tcp://127.0.0.1:49695> has been closed.
13:51:00.509 | INFO    | distributed.batched - Batched Comm Closed <TCP (closed) Scheduler connection to worker local=<tcp://127.0.0.1:49695> remote=<tcp://127.0.0.1:49722>>
Traceback (most recent call last):
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/distributed/batched.py", line 115, in _background_send
    nbytes = yield coro
             ^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/tornado/gen.py", line 766, in run
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/distributed/comm/tcp.py", line 263, in write
    raise CommClosedError()
distributed.comm.core.CommClosedError
13:51:00.512 | INFO    | distributed.batched - Batched Comm Closed <TCP (closed) Scheduler connection to worker local=<tcp://127.0.0.1:49695> remote=<tcp://127.0.0.1:49743>>
Traceback (most recent call last):
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/distributed/batched.py", line 115, in _background_send
    nbytes = yield coro
             ^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/tornado/gen.py", line 766, in run
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/Users/joe/Library/Caches/pypoetry/virtualenvs/feature-library-5-bR7ZDt-py3.12/lib/python3.12/site-packages/distributed/comm/tcp.py", line 263, in write
    raise CommClosedError()
distributed.comm.core.CommClosedError
13:51:00.512 | INFO    | distributed.core - Received 'close-stream' from <tcp://127.0.0.1:49730>; closing.
13:51:00.512 | INFO    | distributed.core - Received 'close-stream' from <tcp://127.0.0.1:49746>; closing.
13:51:00.517 | INFO    | distributed.scheduler - Remove worker addr: <tcp://127.0.0.1:49724> name: 10 (stimulus_id='handle-worker-cleanup-1738090260.5171509')
13:51:00.517 | INFO    | distributed.scheduler - Remove worker addr: <tcp://127.0.0.1:49737> name: 9 (stimulus_id='handle-worker-cleanup-1738090260.5175948')
13:51:00.517 | INFO    | distributed.scheduler - Lost all workers