Has anyone seen an error like this. Don't really k...
# prefect-communityv
Vincent
02/04/2021, 8:00 PMHas anyone seen an error like this. Don't really know where to start debugging for this.
Copy code
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'type': 'system', 'errno': 'EHOSTUNREACH', 'code': 'EHOSTUNREACH'}}}])
Traceback (most recent call last):
File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 98, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 1503, in set_task_run_state
result = self.graphql(
File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'type': 'system', 'errno': 'EHOSTUNREACH', 'code': 'EHOSTUNREACH'}}}]a
Alvis Tang
02/04/2021, 8:09 PMhttp://graphql:443/graphql/? the host doesn't seem to be right
z
Zanie
02/04/2021, 8:17 PM
This is indeed confusing -- can you check on your
config.yomlv
Vincent
02/04/2021, 8:17 PMAgreed - but this error seems to pop up several thousand runs into the flow.
Vincent
02/04/2021, 8:18 PMI am running this on a k8s cluster with prefect cloud.
z
Zanie
02/04/2021, 8:52 PM
Thanks -- we'll look into this.
Zanie
02/04/2021, 10:59 PM
We are continuing to investigate this, these errors shouldn't be surfaced / happening 🙂
v
Vincent
02/12/2021, 6:42 PMHi @Zanie I was wondering if we should open a Bug Report for tracking this issue. This error has popped up and killed my jobs multiple times now.
z
Zanie
02/12/2021, 6:51 PM
Hey, we're having a hard time tracking this down in the backend logs. Do you have timestamps of when this has happened? Cc @Chris White
đź‘€ 1
Zanie
02/12/2021, 6:57 PM
Ah nevermind we did track down some of these. The best guess is that it's a kubedns problem, if you want to open an issue that'd be helpful for tracking it!
v
Vincent
02/12/2021, 6:57 PMI just bumped into one today at 130941
Vincent
02/12/2021, 7:01 PMCan we get the Mavin bot to open up an issue from this thread?
c
Chris White
02/12/2021, 7:05 PM
@Marvin open “Network / API issue when setting task run states?”
m
Marvin
02/12/2021, 7:05 PMs
Shing Fai Chan
03/21/2022, 8:30 AMhi @Vincent @Chris White. I bumped into this issue too. I saw that in the github issue it mentioned it have been fixed, may i know whats the reason and how to fix it ?