Steven Fong
05/14/2022, 8:36 AMprefect.exceptions.ClientError: [{'path': ['create_task_run_artifact'], 'message': 'Task run 4ffb8484-51c9-427e-8604-168dfb707993 not found', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
Anna Geller
prefect diagnostics
?Florian Kühnlenz
05/14/2022, 1:53 PMAnna Geller
Steven Fong
05/14/2022, 4:38 PM296f24ef-ec7f-4fcc-9ada-b626ca92baed
{
"config_overrides": {},
"env_vars": [
"PREFECT__CLOUD__API",
"PREFECT__CLOUD__AGENT__AUTH_TOKEN",
"PREFECT__CLOUD__AGENT__LABELS",
"PREFECT__CLOUD__API_KEY",
"PREFECT__CLOUD__AGENT__AGENT_ADDRESS",
"PREFECT__CLOUD__TENANT_ID",
"PREFECT__BACKEND"
],
"system_information": {
"platform": "Linux-4.14.138-rancher-x86_64-with-glibc2.2.5",
"prefect_backend": "cloud",
"prefect_version": "0.15.5",
"python_version": "3.8.12"
}
}
INFO:prefect.CloudTaskRunner:Task 'run_ge_validation[5]': Starting task run...
Calculating Metrics: 0%| | 0/24 [00:00<?, ?it/s]
Calculating Metrics: 8%|▊ | 2/24 [00:00<00:07, 3.09it/s]
Calculating Metrics: 17%|█▋ | 4/24 [00:00<00:03, 5.45it/s]
Calculating Metrics: 79%|███████▉ | 19/24 [00:01<00:00, 14.81it/s]
Calculating Metrics: 100%|██████████| 24/24 [00:01<00:00, 17.81it/s]
Calculating Metrics: 100%|██████████| 24/24 [00:01<00:00, 14.02it/s]
[2022-05-14 08:00:55+0000] ERROR - prefect.CloudTaskRunner | Task 'run_ge_validation[5]': Exception encountered during task execution!
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 876, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "<string>", line 106, in run_ge_validation
File "/usr/local/lib/python3.9/site-packages/prefect/backend/artifacts.py", line 85, in create_markdown_artifact
return _create_task_run_artifact("markdown", {"markdown": markdown})
File "/usr/local/lib/python3.9/site-packages/prefect/backend/artifacts.py", line 28, in _create_task_run_artifact
return client.create_task_run_artifact(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1836, in create_task_run_artifact
result = self.graphql(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 473, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_task_run_artifact'], 'message': 'Task run 4d86a853-5ac4-4778-9285-ee5287100c97 not found', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
ERROR:prefect.CloudTaskRunner:Task 'run_ge_validation[5]': Exception encountered during task execution!
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 876, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "<string>", line 106, in run_ge_validation
File "/usr/local/lib/python3.9/site-packages/prefect/backend/artifacts.py", line 85, in create_markdown_artifact
return _create_task_run_artifact("markdown", {"markdown": markdown})
File "/usr/local/lib/python3.9/site-packages/prefect/backend/artifacts.py", line 28, in _create_task_run_artifact
return client.create_task_run_artifact(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 1836, in create_task_run_artifact
result = self.graphql(
File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 473, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_task_run_artifact'], 'message': 'Task run 4d86a853-5ac4-4778-9285-ee5287100c97 not found', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
[2022-05-14 08:00:56+0000] INFO - prefect.CloudTaskRunner | Task 'run_ge_validation[5]': Finished task run for task with final state: 'Failed'
Florian Kühnlenz
05/14/2022, 5:41 PM[2022-05-14 14:04:22+0200] ERROR - prefect.CloudTaskRunner | Task 'StartFlowRun[0]': Exception encountered during task execution!
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 876, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 479, in run_task_with_timeout
return run_with_thread_timeout(
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 254, in run_with_thread_timeout
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/tasks.py", line 456, in method
return run_method(self, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/flow_run.py", line 466, in run
create_link_artifact(urlparse(run_link).path)
File "/usr/local/lib/python3.8/site-packages/prefect/backend/artifacts.py", line 52, in create_link_artifact
return _create_task_run_artifact("link", {"link": link})
File "/usr/local/lib/python3.8/site-packages/prefect/backend/artifacts.py", line 28, in _create_task_run_artifact
return client.create_task_run_artifact(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 2160, in create_task_run_artifact
result = self.graphql(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 570, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_task_run_artifact'], 'message': 'Task run 0ca85d4b-dbef-4090-a601-30451c5a9fc8 not found', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
ERROR:prefect.CloudTaskRunner:Task 'StartFlowRun[0]': Exception encountered during task execution!
Flow ID is ccc38835-4618-4852-8a0e-b233a29df1c5Steven Fong
05/14/2022, 6:03 PMFlorian Kühnlenz
05/14/2022, 6:04 PMAnna Geller
Florian Kühnlenz
05/14/2022, 6:26 PMAnna Geller
Steven Fong
05/14/2022, 6:27 PM468ac0af-83de-4c36-8d7f-7a88dc6f2ad2
8b403142-e675-4b5d-9c5c-9f5d07cc732c
e3bd219b-9e63-4f8a-9a32-d968caeb5244
97db0cfd-a7be-493e-8f5f-043990778c64
Anna Geller
Florian Kühnlenz
05/14/2022, 6:31 PMf8426950-0d53-4cea-975d-6d47f05d5e12
This one shows the log from above. I did not yet check the other in detail but I assume similar problems since they did run previously.
265077fa-e8d3-46b1-96d7-88971d5d03bd
5533eda5-138d-4d40-a26b-f13b783ba426
Anna Geller
Steven Fong
05/14/2022, 6:34 PMAnna Geller
Steven Fong
05/14/2022, 6:35 PMAnna Geller
this is what i mean, none of the failures bubble up to the UII see that explains a lot, thanks!
Steven Fong
05/14/2022, 6:36 PMFlorian Kühnlenz
05/14/2022, 6:38 PMSteven Fong
05/14/2022, 6:43 PMAnna Geller
Steven Fong
05/14/2022, 7:24 PMFlorian Kühnlenz
05/14/2022, 7:25 PMSteven Fong
05/14/2022, 7:27 PMcreate_link_artifact(urlparse(run_link).path)
File "/usr/local/lib/python3.8/site-packages/prefect/backend/artifacts.py", line 52, in create_link_artifact
return _create_task_run_artifact("link", {"link": link})
File "/usr/local/lib/python3.8/site-packages/prefect/backend/artifacts.py", line 28, in _create_task_run_artifact
return client.create_task_run_artifact(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 2160, in create_task_run_artifact
result = self.graphql(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 570, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_task_run_artifact'], 'message': 'Task run 0ca85d4b-dbef-4090-a601-30451c5a9fc8 not found', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
its trying to insert the run link into your artifacts.Florian Kühnlenz
05/14/2022, 7:35 PMAnna Geller
Steven Fong
05/15/2022, 3:02 PMFlorian Kühnlenz
05/15/2022, 7:57 PMc3154113-9700-431a-a9dd-10174ee6ce24
Anna Geller
create_flow_run
task to avoid the artifact creation that happens in the StartFlowRun task since it doesn't seem to be an issue in the Cloud API backend anymore - the existing issue got resolved at 1:45 PM UTC and the error in this flow run log was at 6 PM UTC.
Could you try switching to this create_flow_run
task and report back if this helped? (it should)Florian Kühnlenz
05/16/2022, 5:44 AMc3154113-9700-431a-a9dd-10174ee6ce24
is part of our ETL process and should not be run during the day. So we will see how it behaves in the evening. However in the UI it still shows the same message:
Error during execution of task: ClientError([{'path': ['create_task_run_artifact'], 'message': 'Task run 8d67c55d-dc9e-41e3-9993-4d673d0131d2 not found', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
I can not retrieve the logs however.
I also had a look a create_flow_run
and to me it seams this is not a good replacement for StartFlowRun
since it does not wait for the flow run to finish. So we would need to combine it with wait_for_flow_run
. This however is a bit difficult since in many cases we map over StartFlowRun. Any suggestions appreciated.Anna Geller
Florian Kühnlenz
05/17/2022, 6:46 PMd7024b10-e2da-4d5a-bf28-9b6c9ca6cd22
. We did not manage to rewrite the flow yet using the workaround you provided.Anna Geller
create_flow_run
, especially given that this task is much nicer than StartFlowRunFlorian Kühnlenz
05/17/2022, 6:58 PMAnna Geller
Florian Kühnlenz
05/17/2022, 7:04 PM