https://prefect.io logo
#prefect-community
Title
# prefect-community
s

Sander

11/16/2022, 9:32 PM
Hi, I’m working on setting up an ephemeral dask cluster in kubernetes managed by a KubeCluster operator. All orchestrated by prefect. So far so good. But I’m running into some weird error that I don’t find obvious to solve. See thread.
Copy code
21:28:07.434 | INFO    | prefect.task_runner.dask - Creating a new Dask cluster 
with `dask_kubernetes.operator.kubecluster.kubecluster.KubeCluster`
21:28:07.447 | ERROR   | Flow run 'nifty-collie' - Crash detected! Execution was
interrupted by an unexpected exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/task_runners.py", line 
161, in start
    await self._start(exit_stack)
  File "/usr/local/lib/python3.10/site-packages/prefect_dask/task_runners.py", 
line 300, in _start
    self._client = await exit_stack.enter_async_context(
  File "/usr/local/lib/python3.10/contextlib.py", line 619, in 
enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/usr/local/lib/python3.10/site-packages/distributed/client.py", line 
1450, in __aenter__
    await self
  File "/usr/local/lib/python3.10/site-packages/distributed/client.py", line 
1265, in _start
    await self._ensure_connected(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/distributed/client.py", line 
1328, in _ensure_connected
    comm = await connect(
  File "/usr/local/lib/python3.10/site-packages/distributed/comm/core.py", line 
291, in connect
    comm = await asyncio.wait_for(
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 
487, in connect
    ip, port = parse_host_port(address)
  File "/usr/local/lib/python3.10/site-packages/distributed/comm/addressing.py",
line 95, in parse_host_port
    port = _default()
  File "/usr/local/lib/python3.10/site-packages/distributed/comm/addressing.py",
line 73, in _default
    raise ValueError(f"missing port number in address {address!r}")
ValueError: missing port number in address '<Not Connected>'
Does anyone see what could be an issue here? Most stuff runs in default namespace. The dask operator runs in the standard separate namespace.
Does prefect_dask actually work with the new dask operator?
k

Kalise Richmond

11/17/2022, 9:52 PM
Hey @Sander just cross posting here to keep track of the on going conversations https://prefect-community.slack.com/archives/CL09KU1K7/p1668626956875259
👍 1
s

Sander

11/17/2022, 10:02 PM
Hi @Kalise Richmond added some comments in the other thread. Thanks!
3 Views