Thread
#prefect-community
    v

    Vincent

    1 year ago
    Has anyone seen an error like this. Don't really know where to start debugging for this.
    Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'type': 'system', 'errno': 'EHOSTUNREACH', 'code': 'EHOSTUNREACH'}}}])
    Traceback (most recent call last):
      File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 98, in call_runner_target_handlers
        state = self.client.set_task_run_state(
      File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 1503, in set_task_run_state
        result = self.graphql(
      File "/opt/conda/envs/dev/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
        raise ClientError(result["errors"])
    prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'request to <http://graphql:443/graphql/> failed, reason: connect EHOSTUNREACH 10.30.43.11:443', 'type': 'system', 'errno': 'EHOSTUNREACH', 'code': 'EHOSTUNREACH'}}}]
    Alvis Tang

    Alvis Tang

    1 year ago
    http://graphql:443/graphql/? the host doesn't seem to be right
    Michael Adkins

    Michael Adkins

    1 year ago
    This is indeed confusing -- can you check on your
    config.yoml
    ? We'll also need more information on how you're starting your server.
    v

    Vincent

    1 year ago
    Agreed - but this error seems to pop up several thousand runs into the flow.
    I am running this on a k8s cluster with prefect cloud.
    Michael Adkins

    Michael Adkins

    1 year ago
    Thanks -- we'll look into this.
    We are continuing to investigate this, these errors shouldn't be surfaced / happening 🙂
    v

    Vincent

    1 year ago
    Hi @Michael Adkins I was wondering if we should open a Bug Report for tracking this issue. This error has popped up and killed my jobs multiple times now.
    Michael Adkins

    Michael Adkins

    1 year ago
    Hey, we're having a hard time tracking this down in the backend logs. Do you have timestamps of when this has happened? Cc @Chris White
    Ah nevermind we did track down some of these. The best guess is that it's a kubedns problem, if you want to open an issue that'd be helpful for tracking it!
    v

    Vincent

    1 year ago
    I just bumped into one today at 13:09:41
    Can we get the Mavin bot to open up an issue from this thread?
    Chris White

    Chris White

    1 year ago
    @Marvin open “Network / API issue when setting task run states?”
    Marvin

    Marvin

    1 year ago
    Shing Fai Chan

    Shing Fai Chan

    6 months ago
    hi @Vincent @Chris White. I bumped into this issue too. I saw that in the github issue it mentioned it have been fixed, may i know whats the reason and how to fix it ?