m

    Marwan Sarieddine

    1 year ago
    Hi folks, we experienced a flow run failure that seems to stem from prefect cloud given the failure messages we see. Re-running the flow resolves the issue but I thought I would flag it in case it helps.
    We are using prefect v0.14.17 (prefect cloud + kubernetes agent + dask kubernetes) What happens is quite intriguing: All the task runs within the flow fail with the same message:
    Failed to retrieve task state with error: ClientError([{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}])
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
        task_run_info = self.client.get_task_run_info(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1399, in get_task_run_info
        result = self.graphql(mutation)  # type: Any
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
        raise ClientError(result["errors"])
    prefect.utilities.exceptions.ClientError: [{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}]
    The UI seems to crash (I can’t access the schematic and the timeline doesn’t show any of the failed task runs). I am only able to view the logs where I found the error messages. (see the attached screenshots)
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Marwan Sarieddine! This normally indicates the flow registration failed. How did you register your flow?
    m

    Marwan Sarieddine

    1 year ago
    Hi @Kevin Kho - I don’t think that’s case given re-running the flow resolved things
    Kevin Kho

    Kevin Kho

    1 year ago
    Ah that makes sense. Fo you consistently need to re-run and then it works?
    m

    Marwan Sarieddine

    1 year ago
    it just happened once today (this is not a recurring issue)
    and I just re-ran the flow and it resolved things
    Kevin Kho

    Kevin Kho

    1 year ago
    Ah ok. Do let us know if it reoccurs. Is it a big flow?
    m

    Marwan Sarieddine

    1 year ago
    it is a very small flow
    Kevin Kho

    Kevin Kho

    1 year ago
    Ah ok. Well thanks for letting us know 🙂
    Zach Angell

    Zach Angell

    1 year ago
    Hi @Marwan Sarieddine, any chance you could DM me the url to the flow run page screenshot above?
    m

    Marwan Sarieddine

    1 year ago
    sure @Zach Angell