https://prefect.io logo
Join Slack
Powered by
Anybody run into this issue? I'm seeing it on my A...
# ask-community
j
Anybody run into this issue? I'm seeing it on my AWS EC2 instance where I'm running an agent:
Copy code
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
✅ 1
c
Hi Jacob - two things - 1 - would you mind moving the traceback into the thread 2 - you can try setting 
PREFECT_API_ENABLE_HTTP2=False
as an environment variable for the agent
j
Copy code
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/prefect/utilities/services.py", line 46, in critical_service_loop
    await workload()
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/prefect/agent.py", line 194, in get_and_submit_flow_runs
    responses = await self.client.get_scheduled_flow_runs_for_work_pool(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/prefect/client/orchestration.py", line 2290, in get_scheduled_flow_runs_for_work_pool
    response = await <http://self._client.post|self._client.post>(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/prefect/client/base.py", line 227, in send
    response = await self._send_with_retry(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Ok, I can give that a shot. Where should I set the environment variable for the agent? on startup?
On agent startup in the CLI, I mean
c
It should be set however you’d normally set an environment variable for the respective infrastructure - if this is just a local agent, then 
export PREFECT_API_ENABLE_HTTP2=False
; if this is remote somewhere, like Kubernetes, you can set it in the deployment env settings, or as env vars for ACI / ECS
j
I think I can get this into the env var file for the conda environment
c
that’ll work too
I’ve seen this come up a number of times; what version of prefect are you using?
j
lemme check. gotta hop in there and run a check on pip. Has this been fixed recently?
I'm on 2.8.3
c
oh yea
j
Ah ok
Should I just move to the latest version with a pip install --upgrade prefect?
c
generally, yes I would encourage that - this might change your deployment process a bit though, as projects have been reworked to prefect.deployments, although if you are doing it pythonically, it shouldn’t change
j
I had big problems with deploying via python
c
but if you were doing prefect recipe init, that would change
I’d maybe try a 2nd environment and test alongside before upgrading your main one; short term, you can just set / export the environment variable shared earlier
which disables using HTTP2
j
So, if my deployment is pretty vanilla, like this testing example I made: 
prefect deployment build "jacob_simplest_test_flow_ever.py:print_a_message_flow" -sb gitlab-repository/<my repo> -p <my pool> -a --name "v2 of Jacobs gitlab testing flow"
Well, the main environment is cutting out every night and is only relied on for one flow, so I'm just gonna tweak it right there haha
c
that should be available still
j
Ok, nice
c
j
I was just about to ask for that link. Thanks!
c
Those are some of the new changes / features
there’s an interactive mode too, so you can kind of figure it out as you go, and the CLI will prompt you
j
I like this, because the deployment was the most confusing thing in switching from v1 to 2
Good idea
c
Most definitely, and that was a lot of the feedback we received prompting the changes 🙂
j
Thanks again for the help, Christopher!
🙌 1
Ok, so I did the upgrade, then I ran into the same error
So I added that environment variable, but now I'm getting a error saying that my gitlab public key is being denied
Is this a known issue?
c
a gitlab public key? not that I’m aware of, or have heard / seen
j
I've had it pulling from the repo before during testing
c
is it a public repository?
I’m not really exactly sure what the error you’re referencing is here, whether it’s for the agent, or flow runs, or anything
I’m guessing when the flow is trying to clone from the repository, but then you say public key, so is that an ssh public key, or something else?
j
ssh
The machine was configured to connect to the repo via SSH
c
Can you do it manually using that ssh key without prefect?
j
I'll just reset it and try again
c
An SSH key just doesn’t seem like something Prefect would have influence over, unless it was using some sort of block or github credential
j
It's in a block, yeah
c
hrmm, and how are you using that block? Where is it configured to pull from?
or load from
j
Do I have to bring the gitlab credentials into the flow at all?
I just had the deployment pointing to the storage block
Or maybe my ssh agent is off. Shoot, might be that simple
No, it's not that
c
I guess I don’t understand what you actual error is
You just say you your public key is denied, but you aren’t saying when or where?
A picture, some logs?
j
Me neither. Maybe this isn't a prefect error. I can't connect to the gitlab now even just with ssh -T ....
c
gotcha
VPN?
j
I guess I need to eliminate the possibility of issues stemming from potential changes made by our devops to the AWS setup first
It looks like prefect was looking for a particular SSH key
I had to rename the one that had been working successfully before I upgraded prefect yesterday. It works now that I've named it to the default key name "id_rsa"
Open in Slack
PreviousNext
Powered by