Hi folks, we experienced a flow run failure that s...
# prefect-serverm
Marwan Sarieddine
05/10/2021, 8:25 PMHi folks, we experienced a flow run failure that seems to stem from prefect cloud given the failure messages we see. Re-running the flow resolves the issue but I thought I would flag it in case it helps.
Marwan Sarieddine
05/10/2021, 8:26 PMWe are using prefect v0.14.17 (prefect cloud + kubernetes agent + dask kubernetes)
What happens is quite intriguing:
All the task runs within the flow fail with the same message:
Copy code
Failed to retrieve task state with error: ClientError([{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
task_run_info = self.client.get_task_run_info(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1399, in get_task_run_info
result = self.graphql(mutation) # type: Any
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}]Marwan Sarieddine
05/10/2021, 8:27 PMThe UI seems to crash (I can’t access the schematic and the timeline doesn’t show any of the failed task runs). I am only able to view the logs where I found the error messages. (see the attached screenshots)
k
Kevin Kho
05/10/2021, 8:29 PM
Hi @Marwan Sarieddine! This normally indicates the flow registration failed. How did you register your flow?
m
Marwan Sarieddine
05/10/2021, 8:29 PMHi @Kevin Kho - I don’t think that’s case given re-running the flow resolved things
k
Kevin Kho
05/10/2021, 8:31 PM
Ah that makes sense. Fo you consistently need to re-run and then it works?
m
Marwan Sarieddine
05/10/2021, 8:31 PMit just happened once today (this is not a recurring issue)
Marwan Sarieddine
05/10/2021, 8:31 PMand I just re-ran the flow and it resolved things
k
Kevin Kho
05/10/2021, 8:31 PM
Ah ok. Do let us know if it reoccurs. Is it a big flow?
m
Marwan Sarieddine
05/10/2021, 8:31 PMit is a very small flow
k
Kevin Kho
05/10/2021, 8:32 PM
Ah ok. Well thanks for letting us know 🙂
z
Zach Angell
05/10/2021, 8:32 PM
Hi @Marwan Sarieddine, any chance you could DM me the url to the flow run page screenshot above?
m
Marwan Sarieddine
05/10/2021, 8:32 PMsure @Zach Angell
🙏 1