Hi All
We have an issue in our dev prefect suddenly and the issue shows
hasura_1 | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.21.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.ZombieKiller", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.task_run', 'code': 'validation-failed'}, 'message': 'field \"task_run\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n await self.run_once()\n File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 216, in run_once\n await self.reap_zombie_task_runs()\n File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 153, in reap_zombie_task_runs\n apply_schema=False,\n File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n as_box=not apply_schema,\n File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n as_box=as_box,\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.task_run', 'code': 'validation-failed'}, 'message': 'field \"task_run\" not found in type: \\'query_root\\''}]"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.Scheduler", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.flow', 'code': 'validation-failed'}, 'message': 'field \"flow\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n await self.run_once()\n File \"/prefect-server/src/prefect_server/services/towel/scheduler.py\", line 47, in run_once\n offset=500 * iterations,\n File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n as_box=not apply_schema,\n File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n as_box=as_box,\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.flow', 'code': 'validation-failed'}, 'message': 'field \"flow\" not found in type: \\'query_root\\''}]"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n await self.run_once()\n File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 37, in run_once\n return await self.reschedule_flow_runs()\n File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 89, in reschedule_flow_runs\n limit=5000,\n File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n as_box=not apply_schema,\n File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n as_box=as_box,\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}]"}
postgres_1 | 2023-10-26 083307.472 UTC [89] ERROR: insert or update on table "log" violates foreign key constraint "log_flow_run_id_fkey"
postgres_1 | 2023-10-26 083307.472 UTC [89] DETAIL: Key (flow_run_id)=(a40a63bb-a5cd-4a66-bde9-622ef1c13852) is not present in table "flow_run".
Prefect version - 0.15.10