Hey everyone :wave: I'm experience an issue while ...
# prefect-community
s
Hey everyone 👋 I'm experience an issue while starting prefect through the
prefect server start
command. I'm using the 0.15.6 Prefect version and, suddendly, when I try to start the server, an error occurs when hasura tries to create the pgcrypto extension on PostgreSQL. This is the error:
Copy code
postgres_1  | 2022-11-11 15:40:49.418 UTC [84] ERROR:  duplicate key value violates unique constraint "pg_extension_name_index"
postgres_1  | 2022-11-11 15:40:49.418 UTC [84] DETAIL:  Key (extname)=(pgcrypto) already exists.
postgres_1  | 2022-11-11 15:40:49.418 UTC [84] STATEMENT:  CREATE EXTENSION IF NOT EXISTS pgcrypto SCHEMA public
It's strange because I've used prefect until yesterday and I've never had this issue Can anybody help me on this?
I'm also receiving this error:
Copy code
towel_1     | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: [Errno 111] Connect call failed ('172.20.0.3', 3000)", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n    yield\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1385, in post\n    timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1148, in request\n    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1169, in send\n    request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1196, in send_handling_redirects\n    request, auth=auth, timeout=timeout, history=history\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1232, in send_handling_auth\n    response = await self.send_single_request(request, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1269, in send_single_request\n    timeout=timeout.as_dict(),\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 153, in request\n    method, url, headers=headers, stream=stream, timeout=timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 65, in request\n    self.socket = await self._open_socket(timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in _open_socket\n    hostname, port, ssl_context, timeout\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/auto.py\", line 38, in open_tcp_stream\n    return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py\", line 234, in open_tcp_stream\n    stream_reader=stream_reader, stream_writer=stream_writer\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in _exit_\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n    raise to_exc(exc) from None\nhttpcore._exceptions.ConnectError: [Errno 111] Connect call failed ('172.20.0.3', 3000)"}
z
Usually this occurs when multiple servers attempt to start at once.
s
@Zanie I checked it and I don't have multiple Prefect servers trying to start.
These are the other error logs
Copy code
towel_1     | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')", "exc_info": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 164, in try_connect\n    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n  File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1692, in connect_tcp\n    StreamProtocol, host, port, local_addr=local_addr\n  File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n    raise exceptions[0]\n  File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n    await self.sock_connect(sock, address)\n  File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n    return await fut\n  File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n    raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('192.168.32.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n    yield\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 112, in connect_tcp\n    local_host=local_address,\n  File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 222, in connect_tcp\n    raise OSError(\"All connection attempts failed\") from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n    yield\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n    resp = await self._pool.handle_async_request(req)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n    raise exc\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n    response = await connection.handle_async_request(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n    raise exc\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n    stream = await self._connect(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n    stream = await self._network_backend.connect_tcp(**kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 30, in connect_tcp\n    host, port, timeout=timeout, local_address=local_address\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 112, in connect_tcp\n    local_host=local_address,\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n    raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n    yield\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1855, in post\n    extensions=extensions,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1527, in request\n    return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1618, in send\n    history=[],\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1645, in _send_handling_auth\n    history=history,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1679, in _send_handling_redirects\n    response = await self._send_single_request(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1716, in _send_single_request\n    response = await transport.handle_async_request(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n    resp = await self._pool.handle_async_request(req)\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n    raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 37, in run_once\n    return await self.reschedule_flow_runs()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 89, in reschedule_flow_runs\n    limit=500,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 188, in __aexit__\n    await self.gen.athrow(typ, value, traceback)\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 93, in reraise_as_api_error\n    raise APIError() from exc\nprefect_server.utilities.exceptions.APIError: Unable to complete operation. An internal API error occurred."}
Copy code
towel_1     | {"severity": "ERROR", "name": "prefect-server.Scheduler", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')", "exc_info": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 164, in try_connect\n    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n  File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1692, in connect_tcp\n    StreamProtocol, host, port, local_addr=local_addr\n  File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n    raise exceptions[0]\n  File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n    await self.sock_connect(sock, address)\n  File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n    return await fut\n  File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n    raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('192.168.32.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n    yield\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 112, in connect_tcp\n    local_host=local_address,\n  File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 222, in connect_tcp\n    raise OSError(\"All connection attempts failed\") from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n    yield\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n    resp = await self._pool.handle_async_request(req)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n    raise exc\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n    response = await connection.handle_async_request(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n    raise exc\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n    stream = await self._connect(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n    stream = await self._network_backend.connect_tcp(**kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 30, in connect_tcp\n    host, port, timeout=timeout, local_address=local_address\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 112, in connect_tcp\n    local_host=local_address,\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n    raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n    yield\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1855, in post\n    extensions=extensions,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1527, in request\n    return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1618, in send\n    history=[],\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1645, in _send_handling_auth\n    history=history,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1679, in _send_handling_redirects\n    response = await self._send_single_request(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1716, in _send_single_request\n    response = await transport.handle_async_request(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n    resp = await self._pool.handle_async_request(req)\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n    raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/scheduler.py\", line 47, in run_once\n    offset=500 * iterations,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 188, in __aexit__\n    await self.gen.athrow(typ, value, traceback)\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 93, in reraise_as_api_error\n    raise APIError() from exc\nprefect_server.utilities.exceptions.APIError: Unable to complete operation. An internal API error occurred."}
Copy code
towel_1     | {"severity": "ERROR", "name": "prefect-server.ZombieKiller", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')", "exc_info": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 164, in try_connect\n    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n  File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1692, in connect_tcp\n    StreamProtocol, host, port, local_addr=local_addr\n  File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n    raise exceptions[0]\n  File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n    await self.sock_connect(sock, address)\n  File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n    return await fut\n  File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n    raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('192.168.32.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n    yield\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 112, in connect_tcp\n    local_host=local_address,\n  File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 222, in connect_tcp\n    raise OSError(\"All connection attempts failed\") from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n    yield\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n    resp = await self._pool.handle_async_request(req)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n    raise exc\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n    response = await connection.handle_async_request(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n    raise exc\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n    stream = await self._connect(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n    stream = await self._network_backend.connect_tcp(**kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 30, in connect_tcp\n    host, port, timeout=timeout, local_address=local_address\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 112, in connect_tcp\n    local_host=local_address,\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n    raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n    yield\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1855, in post\n    extensions=extensions,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1527, in request\n    return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1618, in send\n    history=[],\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1645, in _send_handling_auth\n    history=history,\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1679, in _send_handling_redirects\n    response = await self._send_single_request(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1716, in _send_single_request\n    response = await transport.handle_async_request(request)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n    resp = await self._pool.handle_async_request(req)\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)\n  File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n    raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 220, in run_once\n    await self.reap_zombie_task_runs()\n  File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 154, in reap_zombie_task_runs\n    apply_schema=False,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n    timeout=30,\n  File \"/usr/local/lib/python3.7/contextlib.py\", line 188, in __aexit__\n    await self.gen.athrow(typ, value, traceback)\n  File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 93, in reraise_as_api_error\n    raise APIError() from exc\nprefect_server.utilities.exceptions.APIError: Unable to complete operation. An internal API error occurred."}
106 Views