https://prefect.io logo
Join Slack
Powered by
Hi everyone, I have encountered this error recentl...
# ask-community
r
Hi everyone, I have encountered this error recently.
Copy code
Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 4b5a98d6-b0c0-4dc5-b540-7f00b17cea2d: provided a running state but associated flow run fc3bdb79-c506-4710-87c1-c039f29d727a is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/cloud/task_runner.py", line 95, in call_runner_target_handlers
    cache_for=self.task.cache_for,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1526, in set_task_run_state
    version=version,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 4b5a98d6-b0c0-4dc5-b540-7f00b17cea2d: provided a running state but associated flow run fc3bdb79-c506-4710-87c1-c039f29d727a is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
It happens when task raised 
FAIL()
and retries. and the the task will be marked 
ClientFailed
state, then the flowrun is 
Cancelled
What I don't understand is why suddenly the error says flowrun not in a running state? How to resolve this?
k
Hey @Ranu Goldan, what version are you on and can you show me the task?
r
Our k8s agent is using 
prefecthq/prefect:0.14.20-python3.6
on deployment
for the task it was about submitting pyspark job to google cloud dataproc with these line at the end:
Copy code
if response_state == "FAILED":
        raise FAIL("Job Failed")
    else:
        status = (
            SUCCESS("Job Success")
            if report_success
            else FAIL("sending add_table_to_data_sources  failed")
        )
        raise status
k
Did this happen just once? Or does this happen every time?
r
I think it happened everytime since 1-2 weeks, with condition that it failed and retries, I also suspects that I have to upgrade the agent image version
udpate: I've tried to upgrade to latest prefect on the agent, but still happens
update: Sorry it was the code in task, some exception handled as
Copy code
prefect_client.set_flow_run_state(prefect.context["flow_run_id"], Cancelled())
thxx @Kevin Kho
👍 1
46 Views
Open in Slack
PreviousNext
Powered by