https://prefect.io
Join Slack
Hi folks, we have been getting unexpected flow run failures (see the thread for more details)
# ask-community
m
Hi folks, we have been getting unexpected flow run failures (see the thread for more details)
Copy code
Failed to retrieve task state with error: ClientError([{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
    task_run_info = self.client.get_task_run_info(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1399, in get_task_run_info
    result = self.graphql(mutation)  # type: Any
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}]
k
Hey @Marwan Sarieddine, a couple of questions. Is this server or cloud? How big is your flow? Is it a flow that has run successfully before?
m
prefect Cloud - it is a small flow that ran successfully before
k
And there was no re-registration in between?
m
yes I re-registered …
k
This error message can mean a corrupt registration so I’m wondering if it runs successfully sometimes after that latest registration
m
I have tried re-running it several times, I get the same error repeatedly
d
@Marwan Sarieddine did you try to change your flow? Or delete it from prefect UI? I found that registration didn't try again if the flow didn't change
k
Ok so I would try registering again and seeing if the same behavior persists. (I assume you did this but just making sure). It may have been a one off unsuccessful registration. I have also seen this pop up when people combine 
flow.run
and 
flow.register
in the same script. It can also happen when you alter the context. It can also happen if you misplace 
flow.register()
, which happens when some people have functions that do multiple registrations
m
I have tried re-registering and it fails - I don’t have a flow.run call but I have a StartFlowRun task
The strange thing with the failure after re-registering is the schematic on the UI shows the updated flow tasks (I removed a case statement) but the logs show tasks running from the old flow version (still show the case statement failing)
perhaps I will delete the flow and re-register to see if it resolves this
so I deleted the flow, re-registered and I am still getting the “case” tasks fail in the logs when there are no “case” tasks in the flow anymore
k
Ok this issue is quite hard to debug. Will pass this to an engineer
m
hmmm - I think I know what is going on
k
Would love to know lol
m
about to verify
ok so this is a self-inflicted bug - apologies for that I re-registered the flow but I didn’t update the image tag, which is why the schematic showed an updated flow, but the image running on our kubernetes cluster was still using the old flow (because of our image pull policy)
thank you @Kevin Kho @davzucky for the support
k
Great info to have. Thanks for sharing! and you figured that out quickly!
👍 1
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by