I have a flow that has been running in production for a few months without issues. However over the past week or two, we've been seeing this error pretty frequently:

Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID a8c171e6-49fc-4370-90ee-ae9319f5848a: provided a running state but associated flow run 5ab28d3b-0dad-4327-8063-f019e5cbbdd5 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/cloud/task_runner.py", line 119, in call_runner_target_handlers
    cache_for=self.task.cache_for,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1085, in set_task_run_state
    version=version,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 225, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID a8c171e6-49fc-4370-90ee-ae9319f5848a: provided a running state but associated flow run 5ab28d3b-0dad-4327-8063-f019e5cbbdd5 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

it's preceded by an error related to the LocalResultHandler like this (this flow is still on version 0.10.2 of core):

Task 'DQ Test - postgres extractions and test creation[40]': unexpected error while running task: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 271, in run
    state=state, upstream_states=upstream_states
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 590, in get_task_inputs
    handler
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/result/base.py", line 294, in to_result
    value = self.result_handler.read(self.value)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/result_handlers/local_result_handler.py", line 62, in read
    with open(fpath, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/src/prefect-result-2020-06-22t04-16-48-214966-00-00'

Any guidance as to what's going on here? Sometimes the flow runs just fine without these errors.

Hey @Mark McDonald, A couple things here: The first error looks to be from initiating a task run (putting it in a Running state) while the flow run itself is in a non-Running state. This is likely caused by your second error, which is attempting to read a file that does not exist. It's also worth noting that the LocalResultHandler will only work if it’s pointed at some shared file system / running non-Dockerized on a single machine. Ideally, using 0.12.0's

LocalResult

as an alternative would be the best course of action, if possible....

thanks @Kyle Moon-Wright - I guess I've been misusing the LocalResultHandler

Cool! It will be well worth the effort switching over to the latest version of Prefect and gaining the newfound functionality of the most recent Results API.