https://prefect.io logo
Join Slack
Powered by
Hello, We have deployed prefect orion on EKS using...
# prefect-kubernetes
u
Hello, We have deployed prefect orion on EKS using helm chart and I am trying to start the agent locally on my laptop by setting prefect config set PREFECT_API_URL=*****, but when I try to start the agent using prefect agent start -q **, I am getting this error RuntimeError: The connection pool was closed while 1 HTTP requests/responses were still in-flight. An exception occurred. Any inputs on this are appreciated.
n
Hi @Uday if applicable, you may want to try this resolution, it seems that this error is usually caused somehow by the setup of the machine attempting to run the agent
👍 1
u
Thanks for the response @Nate I am already on 0.16.2 but still get the same error.
n
hmm okay - do you have a traceback from before this error?
Copy code
RuntimeError: The connection pool was closed while 1 HTTP requests/responses were still in-flight.
u
Copy code
prefect agent start -q agent-test
Starting v2.7.0 agent connected to ***********

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ _|   |   /\ / __| _| \| |   _|
 |  _/   / _|| _|| | (_  | |    / _ \ (_ | _|| .` | | |
 || ||\_|| |__\_| ||   // \\__|_||\| ||


Agent started! Looking for work from queue(s): agent-test...
Traceback (most recent call last):
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/cli/agent.py", line 175, in start
    tg.start_soon(
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 574, in _aexit_
    raise exceptions[0]
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/utilities/services.py", line 46, in critical_service_loop
    await workload()
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/agent.py", line 220, in check_for_cancelled_flow_runs
    cancelling_flow_runs = await self.client.read_flow_runs(
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/client/orion.py", line 1497, in read_flow_runs
    response = await <http://self._client.post|self._client.post>(f"/flow_runs/filter", json=body)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/client/base.py", line 251, in send
    response.raise_for_status()
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/httpx/_models.py", line 745, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Redirect response '302 Found' for url 'https://************/api/flow_runs/filter'
Redirect location: 'https://*******.<http://cloudflareaccess.com/cdn-cgi/access/login/********?kid=**************|cloudflareaccess.com/cdn-cgi/access/login/********?kid=**************>'
For more information check: <https://httpstatuses.com/302>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 205, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 156, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 56, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 233, in run
    return native_run(wrapper(), debug=debug)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper
    return await func(*args)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/cli/agent.py", line 175, in start
    tg.start_soon(
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/agent.py", line 577, in _aexit_
    await self.shutdown(*exc_info)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/agent.py", line 563, in shutdown
    await self.client._aexit_(*exc_info)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/prefect/client/orion.py", line 1898, in _aexit_
    return await self._exit_stack._aexit_(*exc_info)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/contextlib.py", line 670, in _aexit_
    raise exc_details[1]
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/contextlib.py", line 653, in _aexit_
    cb_suppress = await cb(*exc_details)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/httpx/_client.py", line 2003, in _aexit_
    await self.transport._aexit__(exc_type, exc_value, traceback)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/httpx/_transports/default.py", line 332, in _aexit_
    await self.pool._aexit__(exc_type, exc_value, traceback)
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 326, in _aexit_
    await self.aclose()
  File "/Users/Uday.Murikinati/opt/anaconda3/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 312, in aclose
    raise RuntimeError(
RuntimeError: The connection pool was closed while 1 HTTP requests/responses were still in-flight.
An exception occurred.
n
hmm it appears that this is the first exception raised
Copy code
httpx.HTTPStatusError: Redirect response '302 Found' for url 'https://************/api/flow_runs/filter'
Redirect location: 'https://*******.<http://cloudflareaccess.com/cdn-cgi/access/login/********?kid=**************|cloudflareaccess.com/cdn-cgi/access/login/********?kid=**************>'
For more information check: <https://httpstatuses.com/302>
let me see if I can find anything on that
u
Thanks Nate 🙂
n
I'm not familiar with this error, but it seems the issue would likely be related to the 
PREFECT_API_URL
you provided to the helm chart EKS networking isn't necessarily my strong suit, but I wonder if you may need to port forward your local machine ( with your EKS context activated) to the orion deployment like 
kubectl port-forward deployment/orion 4200:4200
(or whichever port you chose) before you try to interact with it from your machine
if that's not helpful, I can ask someone who has some more familiarity with this stuff tomorrow when everyone's back in office
u
Was able to fix it, the issue was with Cloudflare, had to bypass it. Thanks for the help Nate.
👍 1
n
awesome - glad you got it sorted! anytime
Open in Slack
PreviousNext
Powered by