Hey all, Can someone help me explain where this error comes from and how I might resolve it:

Failed to set task state with error: ClientError([{'message': 'State update failed for task run ID 6aa4d54f-d2a5-41d6-9ba8-27ecffe5e209: provided a running state but associated flow run ecb87527-e50d-4ca4-9ed6-4c2f40ef81fe is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 6aa4d54f-d2a5-41d6-9ba8-27ecffe5e209: provided a running state but associated flow run ecb87527-e50d-4ca4-9ed6-4c2f40ef81fe is not in a running state.'}}}])
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
    state = self.client.set_task_run_state(
  File "/opt/conda/lib/python3.8/site-packages/prefect/client/client.py", line 1917, in set_task_run_state
    result = self.graphql(
  File "/opt/conda/lib/python3.8/site-packages/prefect/client/client.py", line 569, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': 'State update failed for task run ID 6aa4d54f-d2a5-41d6-9ba8-27ecffe5e209: provided a running state but associated flow run ecb87527-e50d-4ca4-9ed6-4c2f40ef81fe is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 6aa4d54f-d2a5-41d6-9ba8-27ecffe5e209: provided a running state but associated flow run ecb87527-e50d-4ca4-9ed6-4c2f40ef81fe is not in a running state.'}}}]

We are using Prefect 0.15.2 and have a Dask cluster up and running. Also this, we are receiving multiple apollo timeouts but on a very random basis:

Failed to set task state with error: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='apollo', port=4200): Read timed out. (read timeout=15)"))
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/opt/conda/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/opt/conda/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/opt/conda/lib/python3.8/http/client.py", line 277, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/conda/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

Hey @Fabrice Toussaint, it seems like you are hitting your API heavily? The read timeout can be increased. This is server right?

@Kevin Kho yes it seems that way 😛, where can I increase the read timeout?

Check this and the files changed will show you, but I am not sure it’s the same issue. It should say

Max retries exceeded

.