Hello guys I m trying to figure out why I have this problem Prefect Community #ask-community

Hello guys, I’m trying to figure out why I have t...

Aleksandr Liadov

03/10/2023, 1:08 PM

Hello guys, I’m trying to figure out why I have this problem:

Copy code

socket.gaierror: [Errno -3] Temporary failure in name resolution
<https://api.prefect.cloud/api/accounts/my_account/workspaces/my_workspace/task_runs/0b3215a1-c402-4116-8320-439ca875a59a/set_state> failed

I use last version of prefect and my runner is RayTaskRunner. Infrastructure block is k8s. Any idea? Stacktrace in threads! ``````

Aleksandr Liadov

03/10/2023, 1:09 PM

Copy code

Crash detected! Request to <https://api.prefect.cloud/api/accounts/my_account/workspaces/my_workspace/task_runs/0b3215a1-c402-4116-8320-439ca875a59a/set_state> failed: Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 111, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 189, in connect_tcp
    gai_res = await getaddrinfo(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 496, in getaddrinfo
    gai_res = await get_asynclib().getaddrinfo(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1754, in getaddrinfo
    result = await get_running_loop().getaddrinfo(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 825, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection.py", line 111, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
    return await self._backend.connect_tcp(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 111, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectError: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

httpx.ConnectError: [Errno -3] Temporary failure in name resolution
.
12:43:57 PM
DEBUG
Crash details:
Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 111, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 189, in connect_tcp
    gai_res = await getaddrinfo(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 496, in getaddrinfo
    gai_res = await get_asynclib().getaddrinfo(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1754, in getaddrinfo
    result = await get_running_loop().getaddrinfo(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 825, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_async/connection.py", line 111, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/backends/auto.py", line 29, in connect_tcp
    return await self._backend.connect_tcp(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 111, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ConnectError: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1689, in report_task_run_crashes
    yield
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1328, in begin_task_run
    state = await orchestrate_task_run(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1550, in orchestrate_task_run
    state = await propose_state(client, terminal_state, task_run_id=task_run.id)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1823, in propose_state
    response = await client.set_task_run_state(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/client/orion.py", line 1865, in set_task_run_state
    response = await <http://self._client.post|self._client.post>(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/client/base.py", line 229, in send
    response = await self._send_with_retry(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno -3] Temporary failure in name resolution

5 Views

Open in Slack

Previous Next