Hui Zheng
09/16/2020, 11:02 PMScheduled to start
, but there is not any run logs and no gke job was created. All runs scheduled for the past 3 hours had the same issue. If we manually start a flow-run, it also run into the same issue. The project agent shows in good status. Those delayed runs won’t even show up in the flow runs tab. It seems to me that the flow scheduled the run, but somehow was not able to start the run when the time comes.
[1] https://cloud.prefect.io/semios/flow/dcf941bd-6365-45e8-aafd-71daae6c29f0?version=7nicholas
Hui Zheng
09/16/2020, 11:05 PMnicholas
20.06.0
label to your Agent 🙂Hui Zheng
09/16/2020, 11:06 PM20.06.0
20.06.0
, which is still running in gke cluster, and looks fine
- name: PREFECT__CLOUD__AGENT__LABELS
value: '[''20.06.0'']'
20.06.0
agent disappeared on prefect-cloud dashboard. Right now we only see one other agent,.nicholas
Hui Zheng
09/16/2020, 11:16 PM20.06.0
agent. It is having a lot of errors like this
{
"textPayload": "[2020-09-16 23:10:47,877] ERROR - agent | [{'path': ['get_runs_in_queue'], 'message': '[{\\'extensions\\': {\\'path\\': \\'$\\', \\'code\\': \\'data-exception\\'}, \\'message\\': \\'invalid input syntax for type uuid: \"b1e73cd5-8a84-4c6a-a7ac-69000bd8e827:KubernetesAgent:agent:20.06.0\"\\'}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]\n",
"insertId": "l3nsmegqj3k5xkcd0",
"resource": {
"type": "k8s_container",
"labels": {
"namespace_name": "default",
"location": "us-west2-a",
"project_id": "semios-data-platform",
"pod_name": "prefect-agent-7c8d87df48-ttjpb",
"cluster_name": "scheduler-20-06-0",
"container_name": "agent"
}
},
"timestamp": "2020-09-16T23:10:47.878010266Z",
"severity": "INFO",
"labels": {
"k8s-pod/pod-template-hash": "7c8d87df48",
"k8s-pod/app": "prefect-agent"
},
"logName": "projects/semios-data-platform/logs/stdout",
"receiveTimestamp": "2020-09-16T23:10:54.523562262Z"
}
nicholas
Hui Zheng
09/16/2020, 11:20 PMnicholas
Hui Zheng
09/16/2020, 11:48 PM