https://prefect.io
Join Slack
Has anyone seen an error like this. Don't really know where to start debugging for this. ```Failed t...
# ask-community
v
Has anyone seen an error like this. Don't really know where to start debugging for this.
Copy code
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'type': 'system', 'errno': 'EHOSTUNREACH', 'code': 'EHOSTUNREACH'}}}])
Traceback (most recent call last):
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 98, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 1503, in set_task_run_state
    result = self.graphql(
  File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'type': 'system', 'errno': 'EHOSTUNREACH', 'code': 'EHOSTUNREACH'}}}]
a
http://graphql:443/graphql/? the host doesn't seem to be right
z
This is indeed confusing -- can you check on your 
config.yoml
? We'll also need more information on how you're starting your server.
v
Agreed - but this error seems to pop up several thousand runs into the flow.
I am running this on a k8s cluster with prefect cloud.
z
Thanks -- we'll look into this.
We are continuing to investigate this, these errors shouldn't be surfaced / happening 🙂
v
Hi @Zanie I was wondering if we should open a Bug Report for tracking this issue. This error has popped up and killed my jobs multiple times now.
z
Hey, we're having a hard time tracking this down in the backend logs. Do you have timestamps of when this has happened? Cc @Chris White
đź‘€ 1
Ah nevermind we did track down some of these. The best guess is that it's a kubedns problem, if you want to open an issue that'd be helpful for tracking it!
v
I just bumped into one today at 130941
Can we get the Mavin bot to open up an issue from this thread?
c
@Marvin open “Network / API issue when setting task run states?”
m
s
hi @Vincent @Chris White. I bumped into this issue too. I saw that in the github issue it mentioned it have been fixed, may i know whats the reason and how to fix it ?
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by