i

    Italo Barros

    1 year ago
    Hello everyone, there's a way to configure Prefect so he can use a proxy server? I'm trying to log in on Prefect cloud (using the CLI), but I'm receiving the following SSLError:
    requests.exceptions.SSLError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Italo Barros, just confirming you want to connect to Prefect Cloud behind a proxy?
    i

    Italo Barros

    1 year ago
    @Kevin Kho Yup, that's correct, and also run the Local Agent (with connects with the Cloud) behind the same proxy
    Kevin Kho

    Kevin Kho

    1 year ago
    Will ask the team for any ideas
    Hey @Italo Barros, can you proxy hit other APIs?
    Did you configure anything? And how did you start the agent?
    prefect agent local start
    ?
    Could you try using the request library to hit another API?
    i

    Italo Barros

    1 year ago
    Hi @Kevin Kho, follows the answers: 1 - Yes I'm able to hit other APIs 2 - I didn't configure anything, just installed Prefect and tried to auth on the Prefect Cloud 2 - I didn't start the agent since I was not able to auth, but I will try that and give you feedback asap 3 - I'm using python requests but the module has the option to insert the proxy configurations
    Kevin Kho

    Kevin Kho

    1 year ago
    What happens if you don’t use proxy configurations? Because I think we use requests, but without the configurations.
    i

    Italo Barros

    1 year ago
    I need to use the proxy because I will run the Agent on my company VM located on Azure, and every connection needs to go over the proxy. A strange thing is that I'm able to telnet to api.prefect.io but I'm receiving an SSLError when trying to close the auth connection.
    But answering your question, the auth and agent worked fine on my local computer even using an VPN connection
    Kevin Kho

    Kevin Kho

    1 year ago
    I understand. Just need to make sure this is indeed the issue before we open a feature request for it. what do you mean by close the auth connection? Like stop the agent process?
    i

    Italo Barros

    1 year ago
    @Kevin Kho follows what happens when I try to GET https://api.prefect.io on the server that is behind the proxy (used Insomnia):
    Preparing request to <https://api.prefect.io/>
    * Current time is 2021-08-20T10:28:02.822Z
    * Using libcurl/7.73.0-DEV OpenSSL/1.1.1g zlib/1.2.11 brotli/1.0.9 WinIDN libssh2/1.9.0_DEV nghttp2/1.41.0
    * Using default HTTP version
    * Disable timeout
    * Enable automatic URL encoding
    * Enable SSL validation
    * Enable cookie sending with jar of 0 cookies
    * Enable network proxy for https:
    *   Trying XX.XXX.X.XXX:8080...
    * Connected to <http://proxy.xxxxx.com|proxy.xxxxx.com> (XX.XXX.X.XXX) port 8080 (#0)
    * ALPN, offering http/1.1
    * TLSv1.3 (OUT), TLS handshake, Client hello (1):
    * OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to <http://proxy.xxxxx.com:8080|proxy.xxxxx.com:8080> 
    * Closing connection 0
    But I can run normally on my local computer even with a VPN connection:
    * Preparing request to <https://api.prefect.io/>
    * Current time is 2021-08-20T11:09:19.957Z
    * Using libcurl/7.73.0-DEV OpenSSL/1.1.1g zlib/1.2.11 brotli/1.0.9 WinIDN libssh2/1.9.0_DEV nghttp2/1.41.0
    * Using default HTTP version
    * Disable timeout
    * Enable automatic URL encoding
    * Enable SSL validation
    * Enable cookie sending with jar of 0 cookies
    *   Trying XX.XXX.X.XXX:443...
    * Connected to <http://api.prefect.io|api.prefect.io> (XX.XXX.X.XXX) port 443 (#0)
    * ALPN, offering h2
    * ALPN, offering http/1.1
    * successfully set certificate verify locations:
    *  CAfile: C:\...\Local\Temp\insomnia_2021.4.1\ca-certs.pem
    *  CApath: none
    * TLSv1.3 (OUT), TLS handshake, Client hello (1):
    * TLSv1.3 (IN), TLS handshake, Server hello (2):
    * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
    * TLSv1.3 (IN), TLS handshake, Certificate (11):
    * TLSv1.3 (IN), TLS handshake, CERT verify (15):
    * TLSv1.3 (IN), TLS handshake, Finished (20):
    * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
    * TLSv1.3 (OUT), TLS handshake, Finished (20):
    * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
    * ALPN, server accepted to use h2
    * Server certificate:
    *  subject: CN=<http://api.prefect.io|api.prefect.io>
    *  start date: Jul 30 23:16:41 2021 GMT
    *  expire date: Oct 28 23:16:39 2021 GMT
    *  subjectAltName: host "<http://api.prefect.io|api.prefect.io>" matched cert's "<http://api.prefect.io|api.prefect.io>"
    *  issuer: C=US; O=Let's Encrypt; CN=R3
    *  SSL certificate verify ok.
    * Using HTTP2, server supports multi-use
    * Connection state changed (HTTP/2 confirmed)
    * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
    * Using Stream ID: 1 (easy handle 0x1fde7bbe140)
    
    > GET / HTTP/2
    > Host: <http://api.prefect.io|api.prefect.io>
    > user-agent: insomnia/2021.4.1
    > accept: */*
    
    * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
    * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
    * old SSL session ID is stale, removing
    * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
    
    < HTTP/2 200 
    < server: rhino-core-shield
    < date: Fri, 20 Aug 2021 11:09:20 GMT
    < content-type: text/html; charset=utf-8
    < vary: Accept-Encoding
    < expires: Thu, 01 Jan 1970 00:01:48 GMT
    < cache-control: no-cache, private, no-transform, no-store
    < pragma: no-cache
    < p3p: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
    < via: 1.1 google
    < alt-svc: clear
    I'm also able to properly use the requests module on the server (as in the image attached). I can try to run fiddler to see where the connection is being closed, but I think it's something related to the proxy configuration since I had this issue with git and conda and needed to add the proxy server on their config. I tried to run the "prefect agent local start" and received the following error:
    Traceback (most recent call last):
      File "C:\ProgramData\Anaconda3\lib\site-packages\prefect\agent\agent.py", line 901, in _setup_api_connection
        self._verify_token(self.client.get_auth_token())
      File "C:\ProgramData\Anaconda3\lib\site-packages\prefect\agent\agent.py", line 831, in _verify_token
        raise AuthorizationError("No agent API token provided.")
    I believe the error is related to the fact that I'm unable to Auth with the Prefect Cloud (due to the SSL Error). One strange thing is that I'm able to telnet the api.prefect.io at port 443.
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh what Prefect version are you on?
    If you’re on 0.15.0, just add your
    --key
    to the agent like
    prefect agent local start --key API_KEY_HERE
    to get authenticate
    i

    Italo Barros

    1 year ago
    @Kevin Kho tried to run the agent with the key instead of auth on the Cloud first and still facing the SSLError. Follows the error code:
    Traceback (most recent call last):
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
        self._prepare_proxy(conn)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
        conn.connect()
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connection.py", line 359, in connect
        conn = self._connect_tls_proxy(hostname, conn)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connection.py", line 506, in _connect_tls_proxy
        ssl_context=ssl_context,
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\util\ssl_.py", line 450, in ssl_wrap_socket
        sock, context, tls_in_tls, server_hostname=server_hostname
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\util\ssl_.py", line 493, in _ssl_wrap_socket_impl
        return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\ssl.py", line 407, in wrap_socket
        _context=self, _session=session)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\ssl.py", line 817, in __init__
        self.do_handshake()
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\ssl.py", line 1077, in do_handshake
        self._sslobj.do_handshake()
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\ssl.py", line 689, in do_handshake
        self._sslobj.do_handshake()
    ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:852)
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\requests\adapters.py", line 449, in send
        timeout=timeout
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 796, in urlopen
        **response_kw
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 796, in urlopen
        **response_kw
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 796, in urlopen
        **response_kw
      [Previous line repeated 3 more times]
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\connectionpool.py", line 756, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\urllib3\util\retry.py", line 574, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\Scripts\prefect-script.py", line 9, in <module>
        sys.exit(cli())
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1137, in __call__
        return self.main(*args, **kwargs)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1062, in main
        rv = self.invoke(ctx)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1668, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1668, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1668, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\click\core.py", line 763, in invoke
        return __callback(*args, **kwargs)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\cli\agent.py", line 178, in start
        start_agent(LocalAgent, import_paths=list(import_paths), **kwargs)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\cli\agent.py", line 136, in start_agent
        agent.start()
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\agent\agent.py", line 189, in start
        self._setup_api_connection()
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\agent\agent.py", line 910, in _setup_api_connection
        self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\agent\agent.py", line 858, in _register_agent
        agent_config_id=self.agent_config_id,
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 2024, in register_agent
        tenant_id=self.tenant_id,
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 324, in tenant_id
        self._tenant_id = self._get_auth_tenant()
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 209, in _get_auth_tenant
        response = self.graphql({"query": {"auth_info": "tenant_id"}})
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 549, in graphql
        retry_on_api_error=retry_on_api_error,
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 453, in post
        retry_on_api_error=retry_on_api_error,
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 711, in _request
        session=session, method=method, url=url, params=params, headers=headers
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\prefect\client\client.py", line 600, in _send_request
        timeout=prefect.context.config.cloud.request_timeout,
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\requests\sessions.py", line 590, in post
        return self.request('POST', url, data=data, json=json, **kwargs)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\requests\sessions.py", line 542, in request
        resp = self.send(prep, **send_kwargs)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\requests\sessions.py", line 655, in send
        r = adapter.send(request, **kwargs)
      File "C:\ProgramData\Anaconda3\envs\weekly_rvs\lib\site-packages\requests\adapters.py", line 514, in send
        raise SSLError(e, request=request)
    requests.exceptions.SSLError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))
    Kevin Kho

    Kevin Kho

    1 year ago
    What is your python version btw? I feel like 3.9 is stricter on these things.
    i

    Italo Barros

    1 year ago
    @Kevin Kho I'm using Python 3.6.12 and Prefect 0.15.3
    Kevin Kho

    Kevin Kho

    1 year ago
    Sorry but I am not sure if there’s any way we can help at this time. What I can do though is explore the possibility of adding the proxy configuration to the agent. Could you are how you do it with the requests library so I get a better picture?
    i

    Italo Barros

    1 year ago
    @Kevin Kho, since you are here, I was also trying to bypass this SSL error by deploying the server as a single node on an Azure VM (Windows Server 2019). But I'm unable to start the server because the images are being pulled as Linux containers. There's a way to pull the needed libraries (postgres, hasura, graphql, apollo, tower, ui) as Windows containers instead? I already changed to use Windows containers on Docker desktop and didn't seem to work either.
    Kevin Kho

    Kevin Kho

    1 year ago
    Will ask the team about that
    i

    Italo Barros

    1 year ago
    @Kevin Kho regarding the SSLError, no problem at all! Please let me know if you guys want some help (or another hand) to implement that! Regarding how to use proxy with the requests library, it can be simple as:
    http_proxy  = "<http://10.10.1.10:3128>"
    https_proxy = "<https://10.10.1.11:1080>"
    ftp_proxy   = "<ftp://10.10.1.10:3128>"
    
    proxyDict = { 
                  "http"  : http_proxy, 
                  "https" : https_proxy, 
                  "ftp"   : ftp_proxy
                }
    
    r = requests.get(url, headers=headers, proxies=proxyDict)
    Kevin Kho

    Kevin Kho

    1 year ago
    Thanks I’ll use this if I open a ticket
    i

    Italo Barros

    1 year ago
    Okay, let me know if you need further information from my side. Have a great day!
    Kevin Kho

    Kevin Kho

    1 year ago
    Windows server deployments are not supported
    i

    Italo Barros

    1 year ago
    @Kevin Kho got it, thank you! I will take a look at how to solve the SSL error them
    @Kevin Kho just an update here, I was able to properly auth on Prefect Cloud after updating the package "certifi", running:
    conda update certifi
    The agent now appears on my Prefect cloud and is running okay. Follows the overflow post where I found a similar problem with this solution: https://stackoverflow.com/questions/41691327/ssl-sslerror-ssl-certificate-verify-failed-certificate-verify-failed-ssl-c
    Kevin Kho

    Kevin Kho

    1 year ago
    Wow that’s good to know this thread is too long with Marvin to archive I’ll just archive it myself lol