Hi folks, we have been getting unexpected flow run...
# ask-communitym
Marwan Sarieddine
06/04/2021, 2:12 PMHi folks, we have been getting unexpected flow run failures (see the thread for more details)
Marwan Sarieddine
06/04/2021, 2:13 PM Copy code
Failed to retrieve task state with error: ClientError([{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
task_run_info = self.client.get_task_run_info(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1399, in get_task_run_info
result = self.graphql(mutation) # type: Any
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}]k
Kevin Kho
06/04/2021, 2:14 PM
Hey @Marwan Sarieddine, a couple of questions. Is this server or cloud? How big is your flow? Is it a flow that has run successfully before?
m
Marwan Sarieddine
06/04/2021, 2:14 PMprefect Cloud - it is a small flow that ran successfully before
k
Kevin Kho
06/04/2021, 2:20 PM
And there was no re-registration in between?
m
Marwan Sarieddine
06/04/2021, 2:21 PMyes I re-registered …
k
Kevin Kho
06/04/2021, 2:22 PM
This error message can mean a corrupt registration so I’m wondering if it runs successfully sometimes after that latest registration
m
Marwan Sarieddine
06/04/2021, 2:22 PMI have tried re-running it several times, I get the same error repeatedly
d
davzucky
06/04/2021, 2:26 PM@Marwan Sarieddine did you try to change your flow? Or delete it from prefect UI?
I found that registration didn't try again if the flow didn't change
k
Kevin Kho
06/04/2021, 2:26 PM
Ok so I would try registering again and seeing if the same behavior persists. (I assume you did this but just making sure). It may have been a one off unsuccessful registration. I have also seen this pop up when people combine
flow.runflow.registerflow.register()m
Marwan Sarieddine
06/04/2021, 2:27 PMI have tried re-registering and it fails - I don’t have a flow.run call but I have a StartFlowRun task
Marwan Sarieddine
06/04/2021, 2:27 PMThe strange thing with the failure after re-registering is the schematic on the UI shows the updated flow tasks (I removed a case statement) but the logs show tasks running from the old flow version (still show the case statement failing)
Marwan Sarieddine
06/04/2021, 2:28 PMperhaps I will delete the flow and re-register to see if it resolves this
Marwan Sarieddine
06/04/2021, 2:32 PMso I deleted the flow, re-registered and I am still getting the “case” tasks fail in the logs when there are no “case” tasks in the flow anymore
k
Kevin Kho
06/04/2021, 2:33 PM
Ok this issue is quite hard to debug. Will pass this to an engineer
m
Marwan Sarieddine
06/04/2021, 2:34 PMhmmm - I think I know what is going on
k
Kevin Kho
06/04/2021, 2:35 PM
Would love to know lol
m
Marwan Sarieddine
06/04/2021, 2:35 PMabout to verify
Marwan Sarieddine
06/04/2021, 2:39 PMok so this is a self-inflicted bug - apologies for that
I re-registered the flow but I didn’t update the image tag, which is why the schematic showed an updated flow, but the image running on our kubernetes cluster was still using the old flow (because of our image pull policy)
Marwan Sarieddine
06/04/2021, 2:40 PMthank you @Kevin Kho @davzucky for the support
k
Kevin Kho
06/04/2021, 2:40 PM
Great info to have. Thanks for sharing! and you figured that out quickly!
👍 1
4 Views