I’m seeing following exception irregularly on our prefect agent ec2

Nov 04 18:03:30 ip-172-31-30-84 prefect[242862]: 18:03:30.621 | ERROR   | prefect.agent - Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED
Nov 04 18:03:30 ip-172-31-30-84 prefect[242862]: Traceback (most recent call last):
Nov 04 18:03:30 ip-172-31-30-84 prefect[242862]:   File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
Nov 04 18:03:30 ip-172-31-30-84 prefect[242862]:     func, target_state = self._transitions[(self.state, input_)]
Nov 04 18:03:30 ip-172-31-30-84 prefect[242862]: KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.RECV_PING: 14>)

Full trace in attachment This started to happen as soon as we upgraded from prefect 2.4.0 to 2.6.5 We are using Prefect Cloud. Is this a known issue? 😕

This isn’t a known issue and is pretty weird. This looks like a consequence of us upgrading client connections to use HTTP2 for more efficiency when communicating with Cloud.

I’ve now updated to 2.6.6 to see, if the issue is still present

It kind of looks like an httpx/httpcore bug though

Nothing suspicious before the error.. it seems like some checks from Prefect Cloud fail, but most do not.

Nov 03 10:38:05 ip-172-31-30-84 prefect[242862]: 10:38:05.427 | INFO    | prefect.infrastructure.process - Process 'cinnamon-quoll' exited cleanly.

Nov 03 10:48:31 ip-172-31-30-84 prefect[242862]: 10:48:31.664 | ERROR   | prefect.agent - Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED

👍 I don’t think we’ll get more from debug level logs in our library. If anything we’d need more logs from httpx/httpcore

Can do! I’ll first see if 2.6.6. works any differently and later tonight / early tomorrow will do a github issue

It seems unlikely from the behavior but it’s worth a try 🙂 Thanks!

Can now confirm 2.6.6 is also affected, as you’d suspected 😕 Will open an github issue first thing in the morning. Thanks for the help!

@jpuris any resolution you found?

Unfortunately no 😕 We simply let it happen and hope for an eventual fix 🤞

We’re hoping to have a workaround this week

Upgraded the agent to 2.7.0 and so far logs are clear of

ConnectionInputs.RECV_PING

errors! I do now see this warning every time the flow is about to start.

Dec 02 19:10:00 ip-172-31-30-84 prefect[424753]: 19:10:00.748 | INFO    | prefect.agent - Completed submission of flow run '7c34feec-2e89-4d21-aa95-75b308cfc8b5'
Dec 02 19:10:03 ip-172-31-30-84 prefect[424753]: /usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
Dec 02 19:10:03 ip-172-31-30-84 prefect[424753]:   warn(RuntimeWarning(msg))

$ head -20 /usr/lib/python3.10/runpy.py
"""runpy.py - locating and running Python code using the module namespace

Provides support for locating and running Python scripts using the Python
module namespace instead of the native filesystem.

This allows Python code to play nicely with non-filesystem based PEP 302
importers when locating support scripts as well as when importing modules.
"""
# Written by Nick Coghlan <ncoghlan at gmail.com>
#    to implement PEP 338 (Executing Modules as Scripts)


import sys
import importlib.machinery # importlib first so we can test #15386 via -m
import importlib.util
import io
import types
import os

$ python3 --version
Python 3.10.6

AWS EC2 running 22.04.1 LTS (Jammy Jellyfish) 🤷

I think that warning is safe to ignore, but I would like to fix it