https://prefect.io logo
Title
f

Fabrice Toussaint

05/06/2021, 7:32 AM
Hey everyone, Does anyone know why I get this errors (sometimes) with scheduled tasks?
Failed to retrieve task state with error: ClientError([{'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 5}], 'path': ['get_or_create_task_run_info'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: '}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
    task_run_info = self.client.get_task_run_info(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1399, in get_task_run_info
    result = self.graphql(mutation)  # type: Any
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 5}], 'path': ['get_or_create_task_run_info'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: '}}}]
Our Prefect server is deployed using Kubernetes on GCP with autoscaling enabled, maybe it has something to do with that?
k

Kevin Kho

05/06/2021, 1:34 PM
Hey @Fabrice Toussaint, is your flow very large?
f

Fabrice Toussaint

05/06/2021, 3:20 PM
Eh no, it seems as if this happens when the cluster scales down (during the night) but I am not 100% sure
k

Kevin Kho

05/06/2021, 3:52 PM
The cluster that holds Prefect server scales down?
I’ll ask the team for you. You’re aware Prefect Cloud has 10k task runs for free each month right?
This is a sign that flow registration failed
f

Fabrice Toussaint

05/06/2021, 5:48 PM
@Kevin Kho not the cluster that holds the prefect server but the one that holds the jobs
k

Kevin Kho

05/06/2021, 5:49 PM
Oh I see. Yes this is a sign that flow registration failed. Either something might be off with the Python code or it didn’t register sucessfully.
f

Fabrice Toussaint

05/06/2021, 5:55 PM
The flow is registered beforehand but on a schedule, so maybe there is something off then?
I will check tomorrow if it is still occuring
k

Kevin Kho

05/06/2021, 5:56 PM
Does it work sometimes but fail sometimes? I guess it’s night time where you are? Have a good night 🙂
Will respond when you get back
f

Fabrice Toussaint

05/06/2021, 5:56 PM
Thank you! 😀
@Kevin Kho all three flows which I had scheduled in the night failed
Failed to retrieve task state with error: ClientError([{'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 5}], 'path': ['get_or_create_task_run_info'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: '}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
    task_run_info = self.client.get_task_run_info(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1399, in get_task_run_info
    result = self.graphql(mutation)  # type: Any
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 5}], 'path': ['get_or_create_task_run_info'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: '}}}]
Other flows just keep running fine
0 0 */1 * *
this is the cron schedule I use
Now it is on every flow with a schedule, I have one without where it does not happen
k

Kevin Kho

05/07/2021, 1:44 PM
Could you show how you attach the schedule to the flow?
f

Fabrice Toussaint

05/12/2021, 12:34 PM
@Kevin Kho it seems to only happen to flows with a daily CRON schedule:
clocks.CronClock('0 0 */1 * *')
k

Kevin Kho

05/12/2021, 4:16 PM
Do you use
flow.schedule = ….
?