https://prefect.io logo
j

jpuris

11/04/2022, 6:21 PM
I’m seeing following exception irregularly on our prefect agent ec2
Copy code
Nov 04 18:03:30 ip-172-31-30-84 prefect[242862]: 18:03:30.621 | ERROR   | prefect.agent - Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED
Nov 04 18:03:30 ip-172-31-30-84 prefect[242862]: Traceback (most recent call last):
Nov 04 18:03:30 ip-172-31-30-84 prefect[242862]:   File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
Nov 04 18:03:30 ip-172-31-30-84 prefect[242862]:     func, target_state = self._transitions[(self.state, input_)]
Nov 04 18:03:30 ip-172-31-30-84 prefect[242862]: KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.RECV_PING: 14>)
Full trace in attachment This started to happen as soon as we upgraded from prefect 2.4.0 to 2.6.5 We are using Prefect Cloud. Is this a known issue? 😕
1
z

Zanie

11/04/2022, 6:25 PM
This isn’t a known issue and is pretty weird. This looks like a consequence of us upgrading client connections to use HTTP2 for more efficiency when communicating with Cloud.
j

jpuris

11/04/2022, 6:25 PM
I’ve now updated to 2.6.6 to see, if the issue is still present
z

Zanie

11/04/2022, 6:25 PM
It kind of looks like an httpx/httpcore bug though
Are there any errors before this?
It looks like the complaint is that a closed connection is being used, cc @Zach Angell
j

jpuris

11/04/2022, 6:28 PM
Nothing suspicious before the error.. it seems like some checks from Prefect Cloud fail, but most do not.
Copy code
Nov 03 10:38:05 ip-172-31-30-84 prefect[242862]: 10:38:05.427 | INFO    | prefect.infrastructure.process - Process 'cinnamon-quoll' exited cleanly.

Nov 03 10:48:31 ip-172-31-30-84 prefect[242862]: 10:48:31.664 | ERROR   | prefect.agent - Invalid input ConnectionInputs.RECV_PING in state ConnectionState.CLOSED
I can enable debug logs, if you would like me to (though I’d need to know how)
z

Zanie

11/04/2022, 6:29 PM
👍 I don’t think we’ll get more from debug level logs in our library. If anything we’d need more logs from httpx/httpcore
If you want to open an issue that’d be helpful, I think we’ll need to coordinate with their maintainers.
j

jpuris

11/04/2022, 6:31 PM
Can do! I’ll first see if 2.6.6. works any differently and later tonight / early tomorrow will do a github issue
z

Zanie

11/04/2022, 6:32 PM
It seems unlikely from the behavior but it’s worth a try 🙂 Thanks!
j

jpuris

11/04/2022, 7:59 PM
Can now confirm 2.6.6 is also affected, as you’d suspected 😕 Will open an github issue first thing in the morning. Thanks for the help!
🙏 1
For reference, the related github issue
j

Jared Noynaert

11/29/2022, 10:29 PM
@jpuris any resolution you found?
j

jpuris

11/30/2022, 6:28 AM
Unfortunately no 😕 We simply let it happen and hope for an eventual fix 🤞
😅 1
z

Zanie

11/30/2022, 3:57 PM
We’re hoping to have a workaround this week
🚀 1
It looks like a bug in the h2 library that is way beyond my http knowledge to fix 🙂
j

jpuris

12/03/2022, 8:06 AM
Upgraded the agent to 2.7.0 and so far logs are clear of
ConnectionInputs.RECV_PING
errors! I do now see this warning every time the flow is about to start.
Copy code
Dec 02 19:10:00 ip-172-31-30-84 prefect[424753]: 19:10:00.748 | INFO    | prefect.agent - Completed submission of flow run '7c34feec-2e89-4d21-aa95-75b308cfc8b5'
Dec 02 19:10:03 ip-172-31-30-84 prefect[424753]: /usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
Dec 02 19:10:03 ip-172-31-30-84 prefect[424753]:   warn(RuntimeWarning(msg))
Copy code
$ head -20 /usr/lib/python3.10/runpy.py
"""runpy.py - locating and running Python code using the module namespace

Provides support for locating and running Python scripts using the Python
module namespace instead of the native filesystem.

This allows Python code to play nicely with non-filesystem based PEP 302
importers when locating support scripts as well as when importing modules.
"""
# Written by Nick Coghlan <ncoghlan at gmail.com>
#    to implement PEP 338 (Executing Modules as Scripts)


import sys
import importlib.machinery # importlib first so we can test #15386 via -m
import importlib.util
import io
import types
import os
Copy code
$ python3 --version
Python 3.10.6
AWS EC2 running 22.04.1 LTS (Jammy Jellyfish) 🤷
z

Zanie

12/05/2022, 3:28 PM
I think that warning is safe to ignore, but I would like to fix it
I find it incredibly nebulous and unclear 😄
3 Views