Seeing an internal server error I haven't seen before when running my flow related to a missing UUID. I'm running this using a local agent with DaskCloudProvider on ECS

Failed to retrieve task state with error: ClientError([{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
    task_run_info = self.client.get_task_run_info(
  File "/usr/local/lib/python3.8/dist-packages/prefect/client/client.py", line 1399, in get_task_run_info
    result = self.graphql(mutation)  # type: Any
  File "/usr/local/lib/python3.8/dist-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}]

Hey @Andrew Hannigan! Is your flow code too large to share?

It is unfortunately

A quick search in the slack showed this error came when someone defined their flow twice in the and script when they registered

Definitely only getting one flow url output back after the script runs

But you were able to register it previously?

Yes

Well glad it’s running…I suppose just ping again if a similar issue comes up. 😅

Only change was removing the archived flows, it's a pretty large flow so maybe that had something to do with it

That could well be the case

@Kevin Kho Hmm happening again

Failed to retrieve task state with error: ClientError([{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
    task_run_info = self.client.get_task_run_info(
  File "/usr/local/lib/python3.8/dist-packages/prefect/client/client.py", line 1399, in get_task_run_info
    result = self.graphql(mutation)  # type: Any
  File "/usr/local/lib/python3.8/dist-packages/prefect/client/client.py", line 319, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 101}], 'path': None}}}]

Yeah this is beyond me so I’ll ask the team. I may have to get back to you tomorrow for this,

Hey @Andrew Hannigan, a couple pieces of info might be helpful in tracking this down. • What version of Prefect Core are you using? • Are you using Cloud or Server? • How are you registering the flow? (e.g.

flow.register()

, cli command, etc) • How are you running the flow?

Hi @Zach Angell : • 0.14.17 • Cloud • flow.register() • Running via prefect cloud on a local agent. Local agent spins up a dask cluster using ECSCluster class from DaskCloudProvider. All of it runs on ECS.

Thanks for the additional info! I'll do some digging today. Are you still seeing intermittent issues?

@Zach Angell Flow was triggered manually when the error occurree, and I am using s3storage

@Andrew Hannigan we're still looking into this issue on our side, I'll keep you up to date. If you do see the error again, could you DM the link to the flow run?

Yes will do

Thank you!

@Zach Angell I have a run that returned the same error again: “brown-koel”

Quick update, it seems like one of the following may be happening • this line doesn’t have any task runs to iterate over: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/cloud/flow_runner.py#L395 • OR context is being wiped somewhere in between the main client and the dask worker that runs the task Any chance you're doing context manipulation in your flow?

I'm not doing any context manipulation in the flow. These are the only lines in my code that touch the context, and they are just getters:

timestamp = prefect.context.get('scheduled_start_time')
    flow_run_id = prefect.context.get('flow_run_id')

Hmmm that might have something to do with it. Can you replicate somewhat consistently with a remote dask cluster? And, to confirm, you've never seen the issue with a local dask cluster?

Nope never with a local dask cluster

Got it. That definitely helps narrow things down. I'll do more testing today specifically with remote dask clusters. Any chance you could share part or all of your

DaskExecutor

configuration for the flow? e.g.

flow.executor = DaskExecutor(
    cluster_class="dask_cloudprovider.aws.FargateCluster",
    cluster_kwargs={"n_workers": 4, "image": "my-prefect-image"},
)