https://prefect.io logo
Join Slack
Powered by
Hello I run flows in dask cluster. How it can be p...
# ask-community
a
Hello I run flows in dask cluster. How it can be possible, the flow state is 
Completed
, orion graph has 
red parts
, and all my subtasks 
crashed
? And in logs I see 
crashed
. Maybe it changes smth: • I submit with options every task and I add str(uuid.uuid4())[:8] before every task name. • error: 
Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: 2ae637b0currency_symbol-26-99893272733d461fba8a3051f42a13a8-1
I have put logger level to debug and so the stack trace: Crash details:
Copy code
Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1689, in report_task_run_crashes
    yield
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1320, in begin_task_run
    connect_error = await client.api_healthcheck()
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/client/orion.py", line 235, in api_healthcheck
    await self._client.get("/health")
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/client/base.py", line 229, in send
    response = await self._send_with_retry(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection.py", line 111, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
    return await self._backend.connect_tcp(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 111, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 189, in connect_tcp
    gai_res = await getaddrinfo(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 496, in getaddrinfo
    gai_res = await get_asynclib().getaddrinfo(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1754, in getaddrinfo
    result = await get_running_loop().getaddrinfo(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 825, in getaddrinfo
    return await self.run_in_executor(
asyncio.exceptions.CancelledError
@Ryan Peden I have seen that you already worked for Dask problem, maybe you can help me, or I can give your more information?
30 Views
Open in Slack
PreviousNext
Powered by