Has anyone ran into this error when running a Flow...
# prefect-community
s
Has anyone ran into this error when running a Flow (Prefect version 0.12.1) on a Kubernetes Agent:
[2020-07-01 13:40:32] DEBUG - prefect.CloudFlowRunner | Failed to retrieve flow state with error: AttributeError("'SSLSocket' object has no attribute 'connection'")
The Flow runs fine on my local machine, and the healthcheck passes, but when it's deployed to K8, it seems the CloudFlowRunner cannot write logs to Prefect Cloud and aborts the execution of the Flow.
j
🤔 I have not, are you using core’s server or Cloud?
s
Prefect Cloud
that error shows up on the Flow Run's container after it pulls the Docker image and attempts to start the Flow Run
The full logs are:
Copy code
[2020-07-01 13:40:32] DEBUG - prefect.CloudFlowRunner | Failed to retrieve flow state with error: AttributeError("'SSLSocket' object has no attribute 'connection'")
[2020-07-01 13:40:32] CRITICAL - CloudHandler | Failed to write log with error: 'SSLSocket' object has no attribute 'connection'
[2020-07-01 13:40:32] CRITICAL - CloudHandler | Failed to write log with error: 'SSLSocket' object has no attribute 'connection'
[2020-07-01 13:40:32] CRITICAL - CloudHandler | Failed to write log with error: 'SSLSocket' object has no attribute 'connection'
[2020-07-01 13:40:32] CRITICAL - CloudHandler | Failed to write log with error: 'SSLSocket' object has no attribute 'connection'
[2020-07-01 13:40:32] CRITICAL - CloudHandler | Unable to write logs to Prefect Cloud
[2020-07-01 13:40:32] CRITICAL - CloudHandler | Unable to write logs to Prefect Cloud
[2020-07-01 13:40:32] CRITICAL - CloudHandler | Unable to write logs to Prefect Cloud
[2020-07-01 13:40:32] CRITICAL - CloudHandler | Unable to write logs to Prefect Cloud
I'm thinking it's a version dependency issue, the Python packages for that Flow Run are:
Copy code
'ujson==3.0.0',
            'requests==2.24.0',
            'pandas==1.0.5',
            'numpy==1.19.0',
            'sqlalchemy==1.3.18',
            'hvac==0.10.4',
            'snowflake-connector-python==2.2.8',
            'snowflake-sqlalchemy==1.2.3',
            'pyarrow==0.17.1',
j
I’m thinking it might be something urllib3 version related
Yeah ^
Any way you can see which version of urllib3 is being installed in that image? I’m seeing it being related to the snowflake connector and a new version was just released a week ago
s
I'll download it locally
👍 1
urllib3==1.25.9
j
Does that differ compared to the one you have installed locally?
s
nope, I'm creating a fresh venv to test again on my local machine
so the versions are the same, but the difference being when I run locally I'm just doing the
flow.run()
while the other is using the CloudRunner
j
Ah yeah so that makes sense as to why it wouldn’t happen locally since it’s not actually calling out to cloud. Two things: 1. You could try doing a cloud run in that venv using local storage + local agent to check your dependencies 2. Open an issue with your dependencies and we can try to recreate
You could also try pinning the snowflake connector
<2.2.8
and see if that has any effect
s
rolling it back to 2.2.7 fixed that issue, I'll log something in GitHub later today (to hopefully catch these types of dependency errors during the healthcheck)
j
Great @Scott Zelenka!