https://prefect.io logo
Powered by
Title
k

Khyaati Jindal

12/01/2022, 5:01 AM
Hi guy, I am running a agent from my ec2 machine , and after a while, the agent throws this error 
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/5c-ed-sumary-telegram-bot/env/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
    queue_runs = await self.client.get_runs_in_work_queue(
  File "/home/ubuntu/5c-ed-sumary-telegram-bot/env/lib/python3.10/site-packages/prefect/client/orion.py", line 763, in get_runs_in_work_queue
    response = await <http://self._client.post|self._client.post>(
  File "/home/ubuntu/5c-ed-sumary-telegram-bot/env/lib/python3.10/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/home/ubuntu/5c-ed-sumary-telegram-bot/env/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ubuntu/5c-ed-sumary-telegram-bot/env/lib/python3.10/site-packages/prefect/client/base.py", line 160, in send
    await super().send(*args, **kwargs)
  File "/home/ubuntu/5c-ed-sumary-telegram-bot/env/lib/python3.10/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/home/ubuntu/5c-ed-sumary-telegram-bot/env/lib/python3.10/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
File "/home/ubuntu/5c-ed-sumary-telegram-bot/env/lib/python3.10/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/ubuntu/5c-ed-sumary-telegram-bot/env/lib/python3.10/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/ubuntu/5c-ed-sumary-telegram-bot/env/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ubuntu/5c-ed-sumary-telegram-bot/env/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Any idea why this is happening ? I have deployed lot of deployments before, and never had such a issue
z

Zanie

12/01/2022, 5:40 AM
Unfortunately this is a low-level bug in the HTTP stack
We’re actively working with maintainers upstream to find the cause and a fix. See https://github.com/PrefectHQ/prefect/issues/7442 for tracking.
k

Khyaati Jindal

12/01/2022, 6:04 AM
Thanks a lot @Zanie, makes sense because we have been encountering the same issue only after we upgraded to 2.6 . Will downgrade the prefect and check.
z

Zanie

12/01/2022, 6:07 AM
We’ll have a workaround in a release tomorrow hopefully
b

Boggdan Barrientos

12/01/2022, 3:42 PM
@Khyaati Jindal Did you noticed that it occurs after one hour of execution?
k

Khyaati Jindal

12/02/2022, 5:59 AM
No @Boggdan Barrientos, so far , it happens as soon as the scheduler reaches the execution time.
Today , after rolling it back to 2.4.1 , I am seeing this issue 
ERROR  | prefect.agent - Server error '500 Internal Server Error' for url
'<https://api.prefect.cloud/api/accounts/id/workspaces/id/work_queues/id/get_runs>'
Response: {'detail': {'exception_message': 'Internal Server Error'}}
For more information check: <https://httpstatuses.com/500>
Traceback (most recent call last):
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
  queue_runs = await self.client.get_runs_in_work_queue(
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/client/orion.py", line 763, in get_runs_in_work_queue
  response = await <http://self._client.post|self._client.post>(
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/httpx/_client.py", line 1848, in post
  return await self.request(
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
  return await self.send(request, auth=auth, follow_redirects=follow_redirects)
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/client/base.py", line 183, in send
  response.raise_for_status()
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/client/base.py", line 125, in raise_for_status
  raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url'<https://api.prefect.cloud/api/accounts/id/workspaces/id/work_queues/id/get_runs>'
Response: {'detail': {'exception_message': 'Internal Server Error'}}
For more information check: <https://httpstatuses.com/500>
Traceback (most recent call last):
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
  return fn(*args, **kwargs)
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 201, in coroutine_wrapper
  return run_async_in_new_loop(async_fn, *args, **kwargs)
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 152, in run_async_in_new_loop
  return anyio.run(partial(__fn, *args, **kwargs))
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
  return asynclib.run(func, *args, **backend_options)
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
  return native_run(wrapper(), debug=debug)
 File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
  return loop.run_until_complete(main)
 File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
  return future.result()
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
  return await func(*args)
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/cli/agent.py", line 154, in start
  await critical_service_loop(
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/utilities/services.py", line 41, in critical_service_loop
  await workload()
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/agent.py", line 144, in get_and_submit_flow_runs
  async for work_queue in self.get_work_queues():
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/agent.py", line 105, in get_work_queues
  work_queue = await self.client.read_work_queue_by_name(name)
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/client/orion.py", line 702, in read_work_queue_by_name
  response = await self._client.get(f"/work_queues/name/{name}")
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/httpx/_client.py", line 1757, in get
  return await self.request(
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
  return await self.send(request, auth=auth, follow_redirects=follow_redirects)
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/client/base.py", line 183, in send
  response.raise_for_status()
 File "/home/ubuntu/5c_sales_telegram_bot/env/lib/python3.10/site-packages/prefect/client/base.py", line 125, in raise_for_status
  raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/id/workspaces/id/work_queues/name/SALES-DAR>'
Response: {'detail': {'exception_message': 'Internal Server Error'}}
For more information check: <https://httpstatuses.com/500>
An exception occurred.
4 Views
#prefect-communityJoin Slack
Powered by