https://prefect.io logo
Title
m

Marwan Sarieddine

05/10/2021, 8:25 PM
Hi folks, we experienced a flow run failure that seems to stem from prefect cloud given the failure messages we see. Re-running the flow resolves the issue but I thought I would flag it in case it helps.
We are using prefect v0.14.17 (prefect cloud + kubernetes agent + dask kubernetes) What happens is quite intriguing: All the task runs within the flow fail with the same message:
Failed to retrieve task state with error: ClientError([{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
    task_run_info = self.client.get_task_run_info(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1399, in get_task_run_info
    result = self.graphql(mutation)  # type: Any
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}]
The UI seems to crash (I can’t access the schematic and the timeline doesn’t show any of the failed task runs). I am only able to view the logs where I found the error messages. (see the attached screenshots)
k

Kevin Kho

05/10/2021, 8:29 PM
Hi @Marwan Sarieddine! This normally indicates the flow registration failed. How did you register your flow?
m

Marwan Sarieddine

05/10/2021, 8:29 PM
Hi @Kevin Kho - I don’t think that’s case given re-running the flow resolved things
k

Kevin Kho

05/10/2021, 8:31 PM
Ah that makes sense. Fo you consistently need to re-run and then it works?
m

Marwan Sarieddine

05/10/2021, 8:31 PM
it just happened once today (this is not a recurring issue)
and I just re-ran the flow and it resolved things
k

Kevin Kho

05/10/2021, 8:31 PM
Ah ok. Do let us know if it reoccurs. Is it a big flow?
m

Marwan Sarieddine

05/10/2021, 8:31 PM
it is a very small flow
k

Kevin Kho

05/10/2021, 8:32 PM
Ah ok. Well thanks for letting us know 🙂
z

Zach Angell

05/10/2021, 8:32 PM
Hi @Marwan Sarieddine, any chance you could DM me the url to the flow run page screenshot above?
m

Marwan Sarieddine

05/10/2021, 8:32 PM
sure @Zach Angell
:thank-you: 1