https://prefect.io logo
Channels
Powered by
v

Vincent

02/04/2021, 8:00 PM
Has anyone seen an error like this. Don't really know where to start debugging for this.
Copy code
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'type': 'system', 'errno': 'EHOSTUNREACH', 'code': 'EHOSTUNREACH'}}}])
Traceback (most recent call last):
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 98, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 1503, in set_task_run_state
    result = self.graphql(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'type': 'system', 'errno': 'EHOSTUNREACH', 'code': 'EHOSTUNREACH'}}}]
a

Alvis Tang

02/04/2021, 8:09 PM
http://graphql:443/graphql/? the host doesn't seem to be right
z

Zanie

02/04/2021, 8:17 PM
This is indeed confusing -- can you check on your 
config.yoml
? We'll also need more information on how you're starting your server.
v

Vincent

02/04/2021, 8:17 PM
Agreed - but this error seems to pop up several thousand runs into the flow.
I am running this on a k8s cluster with prefect cloud.
z

Zanie

02/04/2021, 8:52 PM
Thanks -- we'll look into this.
We are continuing to investigate this, these errors shouldn't be surfaced / happening 🙂
v

Vincent

02/12/2021, 6:42 PM
Hi @Zanie I was wondering if we should open a Bug Report for tracking this issue. This error has popped up and killed my jobs multiple times now.
z

Zanie

02/12/2021, 6:51 PM
Hey, we're having a hard time tracking this down in the backend logs. Do you have timestamps of when this has happened? Cc @Chris White
đź‘€ 1
Ah nevermind we did track down some of these. The best guess is that it's a kubedns problem, if you want to open an issue that'd be helpful for tracking it!
v

Vincent

02/12/2021, 6:57 PM
I just bumped into one today at 130941
Can we get the Mavin bot to open up an issue from this thread?
c

Chris White

02/12/2021, 7:05 PM
@Marvin open “Network / API issue when setting task run states?”
m

Marvin

02/12/2021, 7:05 PM
s

Shing Fai Chan

03/21/2022, 8:30 AM
hi @Vincent @Chris White. I bumped into this issue too. I saw that in the github issue it mentioned it have been fixed, may i know whats the reason and how to fix it ?
Powered by