https://prefect.io logo
Channels
Powered by
j

Jacob Bedard

06/28/2023, 7:13 PM
Anybody run into this issue? I'm seeing it on my AWS EC2 instance where I'm running an agent:
Copy code
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
✅ 1
c

Christopher Boyd

06/28/2023, 7:29 PM
Hi Jacob - two things - 1 - would you mind moving the traceback into the thread 2 - you can try setting 
PREFECT_API_ENABLE_HTTP2=False
as an environment variable for the agent
j

Jacob Bedard

06/28/2023, 7:30 PM
Copy code
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/prefect/utilities/services.py", line 46, in critical_service_loop
    await workload()
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/prefect/agent.py", line 194, in get_and_submit_flow_runs
    responses = await self.client.get_scheduled_flow_runs_for_work_pool(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/prefect/client/orchestration.py", line 2290, in get_scheduled_flow_runs_for_work_pool
    response = await <http://self._client.post|self._client.post>(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/prefect/client/base.py", line 227, in send
    response = await self._send_with_retry(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/ubuntu/anaconda3/envs/prefect2-ob/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Ok, I can give that a shot. Where should I set the environment variable for the agent? on startup?
On agent startup in the CLI, I mean
c

Christopher Boyd

06/28/2023, 7:32 PM
It should be set however you’d normally set an environment variable for the respective infrastructure - if this is just a local agent, then 
export PREFECT_API_ENABLE_HTTP2=False
; if this is remote somewhere, like Kubernetes, you can set it in the deployment env settings, or as env vars for ACI / ECS
j

Jacob Bedard

06/28/2023, 7:32 PM
I think I can get this into the env var file for the conda environment
c

Christopher Boyd

06/28/2023, 7:33 PM
that’ll work too
I’ve seen this come up a number of times; what version of prefect are you using?
j

Jacob Bedard

06/28/2023, 7:34 PM
lemme check. gotta hop in there and run a check on pip. Has this been fixed recently?
I'm on 2.8.3
c

Christopher Boyd

06/28/2023, 7:35 PM
oh yea
j

Jacob Bedard

06/28/2023, 7:36 PM
Ah ok
Should I just move to the latest version with a pip install --upgrade prefect?
c

Christopher Boyd

06/28/2023, 7:40 PM
generally, yes I would encourage that - this might change your deployment process a bit though, as projects have been reworked to prefect.deployments, although if you are doing it pythonically, it shouldn’t change
j

Jacob Bedard

06/28/2023, 7:40 PM
I had big problems with deploying via python
c

Christopher Boyd

06/28/2023, 7:40 PM
but if you were doing prefect recipe init, that would change
I’d maybe try a 2nd environment and test alongside before upgrading your main one; short term, you can just set / export the environment variable shared earlier
which disables using HTTP2
j

Jacob Bedard

06/28/2023, 7:41 PM
So, if my deployment is pretty vanilla, like this testing example I made: 
prefect deployment build "jacob_simplest_test_flow_ever.py:print_a_message_flow" -sb gitlab-repository/<my repo> -p <my pool> -a --name "v2 of Jacobs gitlab testing flow"
Well, the main environment is cutting out every night and is only relied on for one flow, so I'm just gonna tweak it right there haha
c

Christopher Boyd

06/28/2023, 7:42 PM
that should be available still
j

Jacob Bedard

06/28/2023, 7:42 PM
Ok, nice
c

Christopher Boyd

06/28/2023, 7:42 PM
j

Jacob Bedard

06/28/2023, 7:42 PM
I was just about to ask for that link. Thanks!
c

Christopher Boyd

06/28/2023, 7:42 PM
Those are some of the new changes / features
there’s an interactive mode too, so you can kind of figure it out as you go, and the CLI will prompt you
j

Jacob Bedard

06/28/2023, 7:43 PM
I like this, because the deployment was the most confusing thing in switching from v1 to 2
Good idea
c

Christopher Boyd

06/28/2023, 7:44 PM
Most definitely, and that was a lot of the feedback we received prompting the changes 🙂
j

Jacob Bedard

06/28/2023, 7:44 PM
Thanks again for the help, Christopher!
🙌 1
Ok, so I did the upgrade, then I ran into the same error
So I added that environment variable, but now I'm getting a error saying that my gitlab public key is being denied
Is this a known issue?
c

Christopher Boyd

06/29/2023, 3:33 PM
a gitlab public key? not that I’m aware of, or have heard / seen
j

Jacob Bedard

06/29/2023, 3:34 PM
I've had it pulling from the repo before during testing
c

Christopher Boyd

06/29/2023, 3:35 PM
is it a public repository?
I’m not really exactly sure what the error you’re referencing is here, whether it’s for the agent, or flow runs, or anything
I’m guessing when the flow is trying to clone from the repository, but then you say public key, so is that an ssh public key, or something else?
j

Jacob Bedard

06/29/2023, 3:41 PM
ssh
The machine was configured to connect to the repo via SSH
c

Christopher Boyd

06/29/2023, 3:42 PM
Can you do it manually using that ssh key without prefect?
j

Jacob Bedard

06/29/2023, 3:44 PM
I'll just reset it and try again
c

Christopher Boyd

06/29/2023, 3:45 PM
An SSH key just doesn’t seem like something Prefect would have influence over, unless it was using some sort of block or github credential
j

Jacob Bedard

06/29/2023, 3:45 PM
It's in a block, yeah
c

Christopher Boyd

06/29/2023, 3:46 PM
hrmm, and how are you using that block? Where is it configured to pull from?
or load from
j

Jacob Bedard

06/29/2023, 3:48 PM
Do I have to bring the gitlab credentials into the flow at all?
I just had the deployment pointing to the storage block
Or maybe my ssh agent is off. Shoot, might be that simple
No, it's not that
c

Christopher Boyd

06/29/2023, 4:00 PM
I guess I don’t understand what you actual error is
You just say you your public key is denied, but you aren’t saying when or where?
A picture, some logs?
j

Jacob Bedard

06/29/2023, 4:01 PM
Me neither. Maybe this isn't a prefect error. I can't connect to the gitlab now even just with ssh -T ....
c

Christopher Boyd

06/29/2023, 4:01 PM
gotcha
VPN?
j

Jacob Bedard

06/29/2023, 4:01 PM
I guess I need to eliminate the possibility of issues stemming from potential changes made by our devops to the AWS setup first
It looks like prefect was looking for a particular SSH key
I had to rename the one that had been working successfully before I upgraded prefect yesterday. It works now that I've named it to the default key name "id_rsa"
Powered by