Hi Guys, we are having a zombie flow run that we cannot kill: When we enter the flow run in UI it's empty - please take a look to the print screens. In the logs we are receiving

[2022-10-18 14:42:58,717] ERROR - xyz| Failed to query for flow run metadata
Oct 18 16:42:58 datalake01 bash[17042]: Traceback (most recent call last):
Oct 18 16:42:58 datalake01 bash[17042]:  File "/usr/local/lib/python3.8/dist-packages/prefect/agent/agent.py", line 331, in _submit_deploy_flow_run_jobs
Oct 18 16:42:58 datalake01 bash[17042]:    flow_runs = self._get_flow_run_metadata(flow_run_ids)
Oct 18 16:42:58 datalake01 bash[17042]:  File "/usr/local/lib/python3.8/dist-packages/prefect/agent/agent.py", line 691, in _get_flow_run_metadata
Oct 18 16:42:58 datalake01 bash[17042]:    result = self.client.graphql(query)
Oct 18 16:42:58 datalake01 bash[17042]:  File "/usr/local/lib/python3.8/dist-packages/prefect/client/client.py", line 570, in graphql
Oct 18 16:42:58 datalake01 bash[17042]:    raise ClientError(result["errors"])
Oct 18 16:42:58 datalake01 bash[17042]: prefect.exceptions.ClientError: [{'path': ['flow_run', 0, 'id'], 'message': 'Cannot return null for non-nullable field flow_run.id.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

We tried to restart the agent service and rebuild the docker image but this zoombie is still running. We have the flow run id and when we query it the state is running. Do you have any idea how to kill the flow run? Its Prefect 1. cc @Angelika Tarnawa

Hey @Mateusz Pazdzior you should be able to cancel the flow run from the interactive api with this query

mutation setFlowRunStates($input: [set_flow_run_state_input!]!) {
set_flow_run_states(input: {states: $input}) {
  states {
  id
  __typename
  }
  __typename
}
}

with this input in the query variables

{
  "input": {
    "flow_run_id": "flow_run_id",
    "version": 1,
    "state": {
      "type": "Cancelled",
      "message": "marked task run as Cancelled because \"because\""
    }
  }
}

Here's a screenshot for reference, it's referencing task runs but the concept applies the same 😄 if that doesn't work you can send in a request to help@prefect.io 😄

Hello @Mason Menges we tried to run it but it returned the INTERNAL_SERVER_ERROR.

Just updating here, we were able to locate the flow run and remove it. Let us know if everything looks good on your end. There are more details in the support email which we sent a bit earlier. There's an active investigation going on in regard to this as well.