Brad

    Brad

    1 year ago
    Has anyone had TLS issues connecting to an existing coiled cluster?
    TypeError: TLS expects a `ssl_context` argument of type ssl.SSLContext (perhaps check your TLS configuration?)  Instead got None
    @Kevin Kho I’ll create a new thread here
    full stack trace:
    Traceback (most recent call last):
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/prefect/engine/flow_runner.py", line 442, in get_flow_run_state
        with self.check_for_cancellation(), executor.start():
      File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
        return next(self.gen)
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/prefect/executors/dask.py", line 208, in start
        with Client(self.address, **self.client_kwargs) as client:
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/distributed/client.py", line 760, in __init__
        self.start(timeout=timeout)
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/distributed/client.py", line 1006, in start
        sync(self.loop, self._start, **kwargs)
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/distributed/utils.py", line 338, in sync
        raise exc.with_traceback(tb)
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/distributed/utils.py", line 321, in f
        result[0] = yield future
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
        value = future.result()
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/distributed/client.py", line 1096, in _start
        await self._ensure_connected(timeout=timeout)
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/distributed/client.py", line 1153, in _ensure_connected
        comm = await connect(
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/distributed/comm/core.py", line 285, in connect
        comm = await asyncio.wait_for(
      File "/usr/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
        return fut.result()
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/distributed/comm/tcp.py", line 376, in connect
        kwargs = self._get_connect_args(**connection_args)
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/distributed/comm/tcp.py", line 418, in _get_connect_args
        ctx = _expect_tls_context(connection_args)
      File "/home/brad/.cache/pypoetry/virtualenvs/betting-data-xyxHMm3k-py3.9/lib/python3.9/site-packages/distributed/comm/tcp.py", line 350, in _expect_tls_context
        raise TypeError(
    TypeError: TLS expects a `ssl_context` argument of type ssl.SSLContext (perhaps check your TLS configuration?)  Instead got None
    I note from https://docs.coiled.io/user_guide/troubleshooting/tls_ssl_context_error.html#tls-expects-a-ssl-context-error to check my environments are the same; I’ve confirmed they are
    I create a cluster via:
    cluster = coiled.Cluster(
        n_workers=1,
        software="my-env-docker",
        shutdown_on_close=False,
        name="prefect-my-env-docker",
    )
    and then create an executor:
    from prefect.executors import DaskExecutor
    
    coiled_executor = DaskExecutor(cluster.scheduler_address)
    And the versions I’m running:
    python 
    '3.9.5 (default, May 11 2021, 08:20:37) \n[GCC 10.3.0]'
    
    Name: prefect
    Version: 0.14.22
    ---
    Name: dask
    Version: 2021.6.2
    
    ---
    Name: distributed
    Version: 2021.6.2
    I should also note this works fine if I spin up a cluster per flow via
    coiled_executor = DaskExecutor(cluster_class=coiled.Cluster, 
                                   cluster_kwargs=dict(name="prefect-demo", 
                                                       configuration="my-acct/demo-cluster-config"))
    flow_state = flow.run(executor=coiled_executor)
    Kevin Kho

    Kevin Kho

    1 year ago
    Do you have a code snippet for how to spin up your Coiled software environment?
    Brad

    Brad

    1 year ago
    Hey @Kevin Kho, sorry I missed this one. I’m using a private image. Let me try and replicate with a public one ..