https://prefect.io logo
Join Slack
Powered by
Hi I've noticed the following error thrown in our...
# ask-community
d
Hi I've noticed the following error thrown in our agents output. Looks to be a malformed request to the k8s API. I am unfortunately not sure when this started as we do not save logs for our agents (fixing that now).
Copy code
10:09:10.168 | ERROR   | prefect.agent - An error occured while monitoring flow run '7dde2a66-9429-45c8-b93f-8cbf0ee195cd'. The flow run will not be marked as failed, but an issue may have occurred.

... see thread for full stack ...

request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '1e197494-22ca-42a4-a2ae-6a576f50024c', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '56847d98-65de-47d9-86ba-67cef57766df', 'X-Kubernetes-Pf-Prioritylevel-Uid': '53f2ea6d-3bf3-4231-a47a-e164ef55cc2c', 'Date': 'Mon, 06 Mar 2023 10:09:10 GMT', 'Content-Length': '179'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"strconv.ParseInt: parsing \\"31.160724690009374\\": invalid syntax","reason":"BadRequest","code":400}\n'
Suspect it may have been introduced by this change and it is calculating the remaining_time as a float and not an int https://github.com/PrefectHQ/prefect/commit/4766c2eb764a7317d81eb32ccfa457ebe1b199c0
Copy code
10:09:10.168 | ERROR   | prefect.agent - An error occured while monitoring flow run '7dde2a66-9429-45c8-b93f-8cbf0ee195cd'. The flow run will not be marked as failed, but an issue may have occurred.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 484, in _submit_run_and_capture_errors
    result = await infrastructure.run(task_status=task_status)
  File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 308, in run
    status_code = await run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 670, in _watch_job
    for event in watch.stream(
  File "/usr/local/lib/python3.10/site-packages/kubernetes/watch/watch.py", line 163, in stream
    resp = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api/batch_v1_api.py", line 1601, in list_namespaced_job
    return self.list_namespaced_job_with_http_info(namespace, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api/batch_v1_api.py", line 1716, in list_namespaced_job_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 235, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '1e197494-22ca-42a4-a2ae-6a576f50024c', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '56847d98-65de-47d9-86ba-67cef57766df', 'X-Kubernetes-Pf-Prioritylevel-Uid': '53f2ea6d-3bf3-4231-a47a-e164ef55cc2c', 'Date': 'Mon, 06 Mar 2023 10:09:10 GMT', 'Content-Length': '179'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"strconv.ParseInt: parsing \\"31.160724690009374\\": invalid syntax","reason":"BadRequest","code":400}\n'
z
Makes sense!
Interested in contributing a fix?
d
Happy to take a punt at it, very new to python
z
I think we just need a 
math.ceil
call in there to cast the datatype
This is probably going to be a bit of a pain with our tests though
because we’re testing with floats to keep things fast
d
Thanks for doing this. I was trying to work out how to test it.
z
Someone else from our team hopped on it 🙂
6 Views
Open in Slack
PreviousNext
Powered by