Hello everyone, there's a way to configure Prefect so he can use a proxy server? I'm trying to log in on Prefect cloud (using the CLI), but I'm receiving the following SSLError:

requests.exceptions.SSLError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))

Hey @Italo Barros, just confirming you want to connect to Prefect Cloud behind a proxy?

@Kevin Kho Yup, that's correct, and also run the Local Agent (with connects with the Cloud) behind the same proxy

Will ask the team for any ideas

Hi @Kevin Kho, follows the answers: 1 - Yes I'm able to hit other APIs 2 - I didn't configure anything, just installed Prefect and tried to auth on the Prefect Cloud 2 - I didn't start the agent since I was not able to auth, but I will try that and give you feedback asap 3 - I'm using python requests but the module has the option to insert the proxy configurations

What happens if you don’t use proxy configurations? Because I think we use requests, but without the configurations.

I need to use the proxy because I will run the Agent on my company VM located on Azure, and every connection needs to go over the proxy. A strange thing is that I'm able to telnet to api.prefect.io but I'm receiving an SSLError when trying to close the auth connection.

I understand. Just need to make sure this is indeed the issue before we open a feature request for it. what do you mean by close the auth connection? Like stop the agent process?

@Kevin Kho follows what happens when I try to GET https://api.prefect.io on the server that is behind the proxy (used Insomnia):

Preparing request to <https://api.prefect.io/>
* Current time is 2021-08-20T10:28:02.822Z
* Using libcurl/7.73.0-DEV OpenSSL/1.1.1g zlib/1.2.11 brotli/1.0.9 WinIDN libssh2/1.9.0_DEV nghttp2/1.41.0
* Using default HTTP version
* Disable timeout
* Enable automatic URL encoding
* Enable SSL validation
* Enable cookie sending with jar of 0 cookies
* Enable network proxy for https:
*   Trying XX.XXX.X.XXX:8080...
* Connected to <http://proxy.xxxxx.com|proxy.xxxxx.com> (XX.XXX.X.XXX) port 8080 (#0)
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to <http://proxy.xxxxx.com:8080|proxy.xxxxx.com:8080> 
* Closing connection 0

But I can run normally on my local computer even with a VPN connection:

* Preparing request to <https://api.prefect.io/>
* Current time is 2021-08-20T11:09:19.957Z
* Using libcurl/7.73.0-DEV OpenSSL/1.1.1g zlib/1.2.11 brotli/1.0.9 WinIDN libssh2/1.9.0_DEV nghttp2/1.41.0
* Using default HTTP version
* Disable timeout
* Enable automatic URL encoding
* Enable SSL validation
* Enable cookie sending with jar of 0 cookies
*   Trying XX.XXX.X.XXX:443...
* Connected to <http://api.prefect.io|api.prefect.io> (XX.XXX.X.XXX) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: C:\...\Local\Temp\insomnia_2021.4.1\ca-certs.pem
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=<http://api.prefect.io|api.prefect.io>
*  start date: Jul 30 23:16:41 2021 GMT
*  expire date: Oct 28 23:16:39 2021 GMT
*  subjectAltName: host "<http://api.prefect.io|api.prefect.io>" matched cert's "<http://api.prefect.io|api.prefect.io>"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x1fde7bbe140)

> GET / HTTP/2
> Host: <http://api.prefect.io|api.prefect.io>
> user-agent: insomnia/2021.4.1
> accept: */*

* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!

< HTTP/2 200 
< server: rhino-core-shield
< date: Fri, 20 Aug 2021 11:09:20 GMT
< content-type: text/html; charset=utf-8
< vary: Accept-Encoding
< expires: Thu, 01 Jan 1970 00:01:48 GMT
< cache-control: no-cache, private, no-transform, no-store
< pragma: no-cache
< p3p: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
< via: 1.1 google
< alt-svc: clear

I'm also able to properly use the requests module on the server (as in the image attached). I can try to run fiddler to see where the connection is being closed, but I think it's something related to the proxy configuration since I had this issue with git and conda and needed to add the proxy server on their config. I tried to run the "prefect agent local start" and received the following error:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\prefect\agent\agent.py", line 901, in _setup_api_connection
    self._verify_token(self.client.get_auth_token())
  File "C:\ProgramData\Anaconda3\lib\site-packages\prefect\agent\agent.py", line 831, in _verify_token
    raise AuthorizationError("No agent API token provided.")

I believe the error is related to the fact that I'm unable to Auth with the Prefect Cloud (due to the SSL Error). One strange thing is that I'm able to telnet the api.prefect.io at port 443.

Oh what Prefect version are you on?

@Kevin Kho tried to run the agent with the key instead of auth on the Cloud first and still facing the SSLError. Follows the error code:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connection.py", line 359, in connect
    conn = self._connect_tls_proxy(hostname, conn)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connection.py", line 506, in _connect_tls_proxy
    ssl_context=ssl_context,
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\util\ssl_.py", line 450, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\util\ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\ssl.py", line 817, in __init__
    self.do_handshake()
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 796, in urlopen
    **response_kw
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 796, in urlopen
    **response_kw
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 796, in urlopen
    **response_kw
  [Previous line repeated 3 more times]
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\util\retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\Scripts\prefect-script.py", line 9, in <module>
    sys.exit(cli())
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\cli\agent.py", line 178, in start
    start_agent(LocalAgent, import_paths=list(import_paths), **kwargs)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\cli\agent.py", line 136, in start_agent
    agent.start()
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\agent\agent.py", line 189, in start
    self._setup_api_connection()
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\agent\agent.py", line 910, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\agent\agent.py", line 858, in _register_agent
    agent_config_id=self.agent_config_id,
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 2024, in register_agent
    tenant_id=self.tenant_id,
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 324, in tenant_id
    self._tenant_id = self._get_auth_tenant()
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 209, in _get_auth_tenant
    response = self.graphql({"query": {"auth_info": "tenant_id"}})
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 549, in graphql
    retry_on_api_error=retry_on_api_error,
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 453, in post
    retry_on_api_error=retry_on_api_error,
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 711, in _request
    session=session, method=method, url=url, params=params, headers=headers
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 600, in _send_request
    timeout=prefect.context.config.cloud.request_timeout,
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\requests\sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))

What is your python version btw? I feel like 3.9 is stricter on these things.

@Kevin Kho I'm using Python 3.6.12 and Prefect 0.15.3

Sorry but I am not sure if there’s any way we can help at this time. What I can do though is explore the possibility of adding the proxy configuration to the agent. Could you are how you do it with the requests library so I get a better picture?

@Kevin Kho, since you are here, I was also trying to bypass this SSL error by deploying the server as a single node on an Azure VM (Windows Server 2019). But I'm unable to start the server because the images are being pulled as Linux containers. There's a way to pull the needed libraries (postgres, hasura, graphql, apollo, tower, ui) as Windows containers instead? I already changed to use Windows containers on Docker desktop and didn't seem to work either.

Will ask the team about that

@Kevin Kho regarding the SSLError, no problem at all! Please let me know if you guys want some help (or another hand) to implement that! Regarding how to use proxy with the requests library, it can be simple as:

http_proxy  = "<http://10.10.1.10:3128>"
https_proxy = "<https://10.10.1.11:1080>"
ftp_proxy   = "<ftp://10.10.1.10:3128>"

proxyDict = { 
              "http"  : http_proxy, 
              "https" : https_proxy, 
              "ftp"   : ftp_proxy
            }

r = requests.get(url, headers=headers, proxies=proxyDict)

Thanks I’ll use this if I open a ticket

Okay, let me know if you need further information from my side. Have a great day!

Windows server deployments are not supported

@Kevin Kho got it, thank you! I will take a look at how to solve the SSL error them

Wow that’s good to know this thread is too long with Marvin to archive I’ll just archive it myself lol