getting server timeout when running the deployment from cloud ui
10
0941.192 | DEBUG | prefect.agent - Checking for flow runs...
10
0946.444 | DEBUG | prefect.agent - Checking for flow runs...
10
0951.687 | DEBUG | prefect.agent - Checking for flow runs...
10
0956.936 | DEBUG | prefect.agent - Checking for flow runs...
10
1002.188 | DEBUG | prefect.agent - Checking for flow runs...
10
1007.439 | DEBUG | prefect.agent - Checking for flow runs...
10
1012.683 | DEBUG | prefect.agent - Checking for flow runs...
10
1018.003 | DEBUG | prefect.agent - Checking for flow runs...
10
1023.248 | DEBUG | prefect.agent - Checking for flow runs...
10
1028.530 | DEBUG | prefect.agent - Checking for flow runs...
10
1028.709 | INFO | prefect.agent - Submitting flow run '62df0689-3a3d-4bb0-bcaa-a16fb215df67'
10
1029.536 | ERROR | prefect.agent - Flow runner failed to submit flow run '62df0689-3a3d-4bb0-bcaa-a16fb215df67'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/agent.py", line 169, in submit_run
await self.task_group.start(submit_flow_run, flow_run, infrastructure)
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 701, in start
return await future
File "/usr/local/lib/python3.8/site-packages/prefect/infrastructure/submission.py", line 48, in submit_flow_run
return await infrastructure.run(task_status=task_status)
File "/usr/local/lib/python3.8/site-packages/prefect/infrastructure/kubernetes.py", line 201, in run
job_name = await run_sync_in_worker_thread(self._create_job, manifest)
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 56, in run_sync_in_worker_thread
return await
anyio.to_thread.run_sync(call, cancellable=True)
File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 28, in run_sync
return await get_asynclib().run_sync_in_worker_thread(func, *args, cancellable=cancellable,
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 818, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 754, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.8/site-packages/prefect/infrastructure/kubernetes.py", line 408, in _create_job
job = batch_client.create_namespaced_job(self.namespace, job_manifest)
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/batch_v1_api.py", line 210, in create_namespaced_job
return self.create_namespaced_job_with_http_info(namespace, body, **kwargs) # noqa: E501
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/batch_v1_api.py", line 309, in create_namespaced_job_with_http_info
return self.api_client.call_api(
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 391, in request
return
self.rest_client.POST(url,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 275, in POST
return self.request("POST", url,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c94033e6-af19-414c-8c50-72811876a885', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '1133f237-04af-4269-a466-2f5eb69e6670', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e6f943c5-ae55-45d1-ab0e-cb0161b8bd0e', 'Date': 'Wed, 03 Aug 2022 10
1029 GMT', 'Content-Length': '264'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"The POST operation against Job.batch could not be completed at this time, please try again.","reason":"ServerTimeout","details":{"name":"POST","group":"batch","kind":"Job"},"code":500}