https://prefect.io logo
#prefect-community
Title
# prefect-community
h

haris khan

08/03/2022, 10:17 AM
getting server timeout when running the deployment from cloud ui 100941.192 | DEBUG | prefect.agent - Checking for flow runs... 100946.444 | DEBUG | prefect.agent - Checking for flow runs... 100951.687 | DEBUG | prefect.agent - Checking for flow runs... 100956.936 | DEBUG | prefect.agent - Checking for flow runs... 101002.188 | DEBUG | prefect.agent - Checking for flow runs... 101007.439 | DEBUG | prefect.agent - Checking for flow runs... 101012.683 | DEBUG | prefect.agent - Checking for flow runs... 101018.003 | DEBUG | prefect.agent - Checking for flow runs... 101023.248 | DEBUG | prefect.agent - Checking for flow runs... 101028.530 | DEBUG | prefect.agent - Checking for flow runs... 101028.709 | INFO | prefect.agent - Submitting flow run '62df0689-3a3d-4bb0-bcaa-a16fb215df67' 101029.536 | ERROR | prefect.agent - Flow runner failed to submit flow run '62df0689-3a3d-4bb0-bcaa-a16fb215df67' Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/prefect/agent.py", line 169, in submit_run await self.task_group.start(submit_flow_run, flow_run, infrastructure) File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 701, in start return await future File "/usr/local/lib/python3.8/site-packages/prefect/infrastructure/submission.py", line 48, in submit_flow_run return await infrastructure.run(task_status=task_status) File "/usr/local/lib/python3.8/site-packages/prefect/infrastructure/kubernetes.py", line 201, in run job_name = await run_sync_in_worker_thread(self._create_job, manifest) File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 56, in run_sync_in_worker_thread return await anyio.to_thread.run_sync(call, cancellable=True) File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 28, in run_sync return await get_asynclib().run_sync_in_worker_thread(func, *args, cancellable=cancellable, File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 818, in run_sync_in_worker_thread return await future File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 754, in run result = context.run(func, *args) File "/usr/local/lib/python3.8/site-packages/prefect/infrastructure/kubernetes.py", line 408, in _create_job job = batch_client.create_namespaced_job(self.namespace, job_manifest) File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/batch_v1_api.py", line 210, in create_namespaced_job return self.create_namespaced_job_with_http_info(namespace, body, **kwargs) # noqa: E501 File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/batch_v1_api.py", line 309, in create_namespaced_job_with_http_info return self.api_client.call_api( File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 391, in request return self.rest_client.POST(url, File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 275, in POST return self.request("POST", url, File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 234, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (500) Reason: Internal Server Error HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c94033e6-af19-414c-8c50-72811876a885', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '1133f237-04af-4269-a466-2f5eb69e6670', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e6f943c5-ae55-45d1-ab0e-cb0161b8bd0e', 'Date': 'Wed, 03 Aug 2022 101029 GMT', 'Content-Length': '264'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"The POST operation against Job.batch could not be completed at this time, please try again.","reason":"ServerTimeout","details":{"name":"POST","group":"batch","kind":"Job"},"code":500}
prefect 2.0.1
m

Mathijs Carlu

08/03/2022, 10:27 AM
Don't know what you mean by 'deployment from cloud ui', but it seems like the role and/or rolebinding were not (correctly) created. Note that you should change the namespace value in both files to the namespace where your agent is running (which is orion for me)
🙏 1
a

Anna Geller

08/03/2022, 11:19 AM
please can you move code block to the thread?
👍🏻 1
5 Views