https://prefect.io logo
Channels
Powered by
#prefect-server
Title
# prefect-server
w

Wilson Bilkovich

08/27/2021, 6:37 PM
Sorry for mentioning this again, but I’m still grappling with this error. Does this look familiar to anyone? 
ERROR:asyncio:Fatal error on SSL transport
Copy code
% python demo_flow.py 
[2021-08-27 14:32:43-0400] INFO - prefect.S3 | Uploading script ./demo_flow.py to prefect/flows/demo/random-sample-wlb in addepar-research-data
Flow URL: <http://localhost:8080/default/flow/8919b470-d6bb-4522-a843-6fe5491a2ed0>
 └── ID: 74703757-162b-4cc8-a3d6-2d61f13be9d3
 └── Project: demo
 └── Labels: []
[2021-08-27 14:32:46-0400] INFO - prefect.FlowRunner | Beginning Flow run for 'increment a random sample'
[2021-08-27 14:32:46-0400] INFO - prefect.DaskExecutor | Creating a new Dask cluster with `dask_kubernetes.core.KubeCluster`...
Creating scheduler pod on cluster. This may take some time.
[2021-08-27 14:33:07-0400] INFO - prefect.DaskExecutor | The Dask dashboard is available at <http://dask-wilsonbilkovich-f45daff2-7.prefect:8787/status>
INFO:prefect.DaskExecutor:The Dask dashboard is available at <http://dask-wilsonbilkovich-f45daff2-7.prefect:8787/status>
/Users/wilson.bilkovich/.pyenv/versions/3.9.6/envs/addemart/lib/python3.9/site-packages/distributed/client.py:1105: VersionMismatchWarning: Mismatched versions found

+-------------+-----------+-----------+---------+
| Package     | client    | scheduler | workers |
+-------------+-----------+-----------+---------+
| blosc       | None      | 1.10.2    | None    |
| dask        | 2021.08.0 | 2021.08.1 | None    |
| distributed | 2021.08.0 | 2021.08.1 | None    |
| lz4         | None      | 3.1.3     | None    |
+-------------+-----------+-----------+---------+
  warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
[2021-08-27 14:33:26-0400] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
INFO:prefect.FlowRunner:Flow run SUCCESS: all reference tasks succeeded
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x11da37700>
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x11da373d0>
ERROR:asyncio:Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x11dab9b20>, 45.688264346)]']
connector: <aiohttp.connector.TCPConnector object at 0x11da379a0>
ERROR:asyncio:Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x11db50a00>, 45.813355617)]']
connector: <aiohttp.connector.TCPConnector object at 0x11da37ac0>
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x11da62f10>
transport: <_SelectorSocketTransport closing fd=20>
Traceback (most recent call last):
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/selector_events.py", line 918, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/sslproto.py", line 684, in _process_write_backlog
    self._transport.write(chunk)
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/selector_events.py", line 924, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/selector_events.py", line 719, in _fatal_error
    self._force_close(exc)
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/selector_events.py", line 731, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/base_events.py", line 746, in call_soon
    self._check_closed()
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x11dad98e0>
transport: <_SelectorSocketTransport closing fd=17>
Traceback (most recent call last):
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/selector_events.py", line 918, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/sslproto.py", line 684, in _process_write_backlog
    self._transport.write(chunk)
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/selector_events.py", line 924, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/selector_events.py", line 719, in _fatal_error
    self._force_close(exc)
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/selector_events.py", line 731, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/base_events.py", line 746, in call_soon
    self._check_closed()
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
k

Kevin Kho

08/27/2021, 6:42 PM
Can I see the Flow code? I assume it’s just a simple mapped function right?
w

Wilson Bilkovich

08/27/2021, 6:47 PM
Copy code
Traceback (most recent call last):
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/envs/addemart/lib/python3.9/site-packages/dask_kubernetes/utils.py", line 62, in get_external_address_for_scheduler_service
    socket.getaddrinfo(host, port)
  File "/Users/wilson.bilkovich/.pyenv/versions/3.9.6/lib/python3.9/socket.py", line 953, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
… is a variation of the error I’ve also encountered.
but all the env variables everywhere seem to be set, specifying the hosts and urls
k

Kevin Kho

08/27/2021, 6:58 PM
I really can’t find any leads on this. Just a question though, does this persist with the 
prefect
image?
w

Wilson Bilkovich

08/27/2021, 7:35 PM
Let me try
k

Kevin Kho

08/27/2021, 7:35 PM
prefect only has a python 3.8 image i think so you may need to downgrade your python env version
w

Wilson Bilkovich

08/27/2021, 7:37 PM
Luckily it doesn’t get that far; I get the same error with 
prefecthq/prefect:latest
with 
dask-kubernetes
as the sole extra pip package.
k

Kevin Kho

08/27/2021, 7:45 PM
Yeah, I haven’t seen this before unfortunately, and the searches don’t give anything. My (unsatisfying) best advice would be to go down to python 3.8 to maybe find a stable version that way.
w

Wilson Bilkovich

08/27/2021, 8:58 PM
So, I stopped a port-forwarding that was going on, and it seems to have gone away? I went back to using. 
daskdev/dask
to get the python 3.9 support and it worked
It was just me forwarding a port to Kibana though, so I don’t know how it manages to interfere
k

Kevin Kho

08/27/2021, 9:27 PM
I see. Portforwarding on the dask side?
w

Wilson Bilkovich

08/28/2021, 6:14 PM
Actually just some separate port-forwarding so I could watch the cluster logs at the same time, which is why it puzzles me so much.
k

Kevin Kho

08/28/2021, 6:14 PM
My god I wonder how you even figured it out 😅
w

Wilson Bilkovich

08/28/2021, 6:16 PM
Once, the port-forwarding to Dask failed, and I decided to try canceling it.. could have taken another day to figure out easily haha
2 Views
Powered by