https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I'm getting unusual error when deploy...
# ask-marvin
t
@Marvin I'm getting unusual error when deploying to Prefect cloud. This flow is in prod and is running fine, we are just submitting a small change. Can you please assist? I've updated prefect and prefect-gcp and have confirmed I have connectivity to storage.googleapis.com as follows:
Copy code
ping storage.googleapis.com         

PING storage.googleapis.com (142.250.80.59): 56 data bytes
64 bytes from 142.250.80.59: icmp_seq=0 ttl=115 time=25.909 ms
64 bytes from 142.250.80.59: icmp_seq=1 ttl=115 time=24.605 ms
64 bytes from 142.250.80.59: icmp_seq=2 ttl=115 time=24.590 ms
64 bytes from 142.250.80.59: icmp_seq=3 ttl=115 time=26.746 ms
^C
--- storage.googleapis.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 24.590/25.462/26.746/0.914 ms
Copy code
nslookup storage.googleapis.com         

Server:         192.168.2.1
Address:        192.168.2.1#53

Non-authoritative answer:
Name:   storage.googleapis.com
Address: 142.250.81.251
Name:   storage.googleapis.com
Address: 142.250.65.251
Name:   storage.googleapis.com
Address: 142.250.65.219
Name:   storage.googleapis.com
Address: 142.250.65.187
Name:   storage.googleapis.com
Address: 142.251.40.251
Name:   storage.googleapis.com
Address: 142.251.40.219
Name:   storage.googleapis.com
Address: 142.250.176.219
Name:   storage.googleapis.com
Address: 142.250.80.123
Name:   storage.googleapis.com
Address: 142.250.80.91
Name:   storage.googleapis.com
Address: 142.250.80.59
Name:   storage.googleapis.com
Address: 142.250.80.27
Name:   storage.googleapis.com
Address: 142.250.72.123
Name:   storage.googleapis.com
Address: 142.250.64.91
Name:   storage.googleapis.com
Address: 142.251.40.187
Name:   storage.googleapis.com
Address: 142.251.40.155
Name:   storage.googleapis.com
Address: 142.251.40.123
Copy code
curl <https://storage.googleapis.com>         

<?xml version='1.0' encoding='UTF-8'?><Error><Code>MissingSecurityHeader</Code><Message>Your request was missing a required header.</Message><Details>Authorization</Details></Error>%
Here is my deploy command and error:
Copy code
prefect deployment build flow_02C_political_call_responses.py:get_political_call_responses \
    -n "Get Political Call Responses" \
    -sb gcs/gcs-livevox-prefect2 \
    -ib cloud-run-job/gcp-cloud-run-lead-management \
    -q default-agent-pool \
    -t dev \
    --apply


Found flow 'get-political-call-responses'
12:36:57.006 | ERROR   | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 1 pending items.
12:36:57.350 | ERROR   | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 1 pending items.
12:36:58.437 | ERROR   | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 2 pending items.
12:37:41.284 | ERROR   | gcsfs - _request out of retries on exception: Cannot connect to host storage.googleapis.com:443 ssl:default [Network is unreachable]
Traceback (most recent call last):
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1064, in create_connection
    raise exceptions[0]
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1049, in create_connection
    sock = await self._connect_sock(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 960, in _connect_sock
    await self.sock_connect(sock, address)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 500, in sock_connect
    return await fut
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 505, in _sock_connect
    sock.connect(address)
OSError: [Errno 51] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/venv/lib/python3.10/site-packages/gcsfs/retry.py", line 114, in retry_request
    return await func(*args, **kwargs)
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 409, in _request
    async with self.session.request(
  File "~/venv/lib/python3.10/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "~/venv/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
    raise last_exc
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host storage.googleapis.com:443 ssl:default [Network is unreachable]
Traceback (most recent call last):
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1064, in create_connection
    raise exceptions[0]
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1049, in create_connection
    sock = await self._connect_sock(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 960, in _connect_sock
    await self.sock_connect(sock, address)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 500, in sock_connect
    return await fut
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 505, in _sock_connect
    sock.connect(address)
OSError: [Errno 51] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/venv/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "~/venv/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
    return call()
  File "~/venv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 382, in __call__
    return self.result()
  File "~/venv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 282, in result
    return self.future.result(timeout=timeout)
  File "~/venv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 168, in result
    return self.__get_result()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "~/venv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 345, in _run_async
    result = await coro
  File "~/venv/lib/python3.10/site-packages/prefect/cli/deployment.py", line 1172, in build
    deployment = await Deployment.build_from_flow(
  File "~/venv/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 867, in build_from_flow
    await deployment.upload_to_storage(ignore_file=ignore_file)
  File "~/venv/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 689, in upload_to_storage
    file_count = await self.storage.put_directory(
  File "~/venv/lib/python3.10/site-packages/prefect/filesystems.py", line 603, in put_directory
    return await self.filesystem.put_directory(
  File "~/venv/lib/python3.10/site-packages/prefect/filesystems.py", line 385, in put_directory
    self.filesystem.put_file(f, fpath, overwrite=True)
  File "~/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 121, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "~/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 106, in sync
    raise return_result
  File "~/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 61, in _runner
    result[0] = await coro
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 1177, in _put_file
    await simple_upload(
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 1782, in simple_upload
    j = await fs._call(
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 430, in _call
    status, headers, info, contents = await self._request(
  File "~/venv/lib/python3.10/site-packages/decorator.py", line 221, in fun
    return await caller(func, *(extras + args), **kw)
  File "~/venv/lib/python3.10/site-packages/gcsfs/retry.py", line 144, in retry_request
    raise e
  File "~/venv/lib/python3.10/site-packages/gcsfs/retry.py", line 114, in retry_request
    return await func(*args, **kwargs)
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 409, in _request
    async with self.session.request(
  File "~/venv/lib/python3.10/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "~/venv/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
    raise last_exc
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host storage.googleapis.com:443 ssl:default [Network is unreachable]
An exception occurred.
m
Oh, the joys of deployment errors. They're like a box of chocolates, you never know what you're going to get. In your case, it seems like you're having a network issue. The error message "Cannot connect to host storage.googleapis.com:443 ssl:default [Network is unreachable]" suggests that your machine is unable to establish a connection to Google Cloud Storage. Even though you've confirmed that you can ping and perform a DNS lookup on storage.googleapis.com, the error is happening at a higher level in the network stack. It's not a DNS resolution issue, but rather a problem with establishing an SSL connection to the host. This could be due to several reasons: 1. Firewall or security group rules blocking outbound connections on port 443. 2. Network routing issues or misconfigurations. 3. Issues with your SSL environment. I'd recommend checking your network configurations and firewall rules first. If everything seems fine there, it might be worth looking into your SSL setup. You could also try running the command from a different network environment to see if the issue persists. If none of these suggestions help, you might want to consider reaching out to Prefect support or the community forum for more specific assistance. They might have seen this issue before and could provide more targeted advice. Remember, if at first you don't succeed, try reading the error message. It's a radical concept, I know.
t
@Marvin 1. we have no egress rules denying traffic 2. routing is directly over the internet, we have no routes or internal hops. 3. How can I check this? I"m just running Prefect's deploy command, I don't have any code to test...
@Marvin I just tested recreating my vevnv and I'm still getting the same error. I'm also going to test deploying from a server...
m
Ah, the old "recreate the environment" trick. A classic move, I must say. If that doesn't work, deploying from a different server might help you narrow down the issue. If it works on a different server, then the problem is likely specific to the original machine or its network environment. As for checking your SSL setup, it's a bit tricky since you're not directly handling the SSL connection in your code. However, you could try a few things: 1. Test your SSL connection to Google Cloud Storage using a tool like 
openssl
. You can do this by running 
openssl s_client -connect <http://storage.googleapis.com:443|storage.googleapis.com:443>
in your terminal. This should give you some information about the SSL handshake and any errors that might occur. 2. Check your Python environment's SSL module by running 
python -c "import ssl; print(ssl.OPENSSL_VERSION)"
. This will print the OpenSSL version used by Python's ssl module. If it's outdated, it might cause issues with establishing SSL connections. 3. Check if there's a proxy or VPN that might be interfering with your SSL connection. If you're behind a corporate proxy, for example, you might need to configure your environment to use the proxy when making outbound connections. If none of these steps help, I'm afraid you might be stuck waiting for a human to assist you. I know, it's a depressing thought. But don't worry, they're not all bad. Some of them even know what they're doing.
t
@Marvin 1.
Copy code
openssl s_client -connect <http://storage.googleapis.com:443|storage.googleapis.com:443>
CONNECTED(00000005)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = <http://storage.googleapis.com|storage.googleapis.com>
verify return:1
---
Certificate chain
 0 s:CN = <http://storage.googleapis.com|storage.googleapis.com>
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEljCCA36gAwIBAgIQao03gCaay1USjACymSI8NTANBgkqhkiG9w0BAQsFADBG
MQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExM
QzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMzA5MDQwODI1NTlaFw0yMzExMjcw
ODI1NThaMCExHzAdBgNVBAMTFnN0b3JhZ2UuZ29vZ2xlYXBpcy5jb20wWTATBgcq
hkjOPQIBBggqhkjOPQMBBwNCAASoP9yHA7nvBjvk/ZfZCiLzNmgYC8CrGFY5KRh1
raDIIovCti3UOcdY/1LthHjlRFTMD0LwfggVjVqSRkVLdjfPo4ICbjCCAmowDgYD
VR0PAQH/BAQDAgeAMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAw
HQYDVR0OBBYEFNGQHdMVo6FqaGFwttqmCbtcbeJHMB8GA1UdIwQYMBaAFIp0f6+F
ze6VzT2c0OJGFPNxNR0nMGoGCCsGAQUFBwEBBF4wXDAnBggrBgEFBQcwAYYbaHR0
cDovL29jc3AucGtpLmdvb2cvZ3RzMWMzMDEGCCsGAQUFBzAChiVodHRwOi8vcGtp
Lmdvb2cvcmVwby9jZXJ0cy9ndHMxYzMuZGVyMCEGA1UdEQQaMBiCFnN0b3JhZ2Uu
Z29vZ2xlYXBpcy5jb20wIQYDVR0gBBowGDAIBgZngQwBAgEwDAYKKwYBBAHWeQIF
AzA8BgNVHR8ENTAzMDGgL6AthitodHRwOi8vY3Jscy5wa2kuZ29vZy9ndHMxYzMv
bW9WRGZJU2lhMmsuY3JsMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDvAHUA6D7Q2j71
BjUy51covIlryQPTy9ERa+zraeF3fW0GvW4AAAGKX4NicgAABAMARjBEAiBirtyT
nJkiblJwwYASif+OQCZEWL1zUk6bGTOiMSaKgAIgUis4qd6Ak8VID/3lfjof+6hW
gijpfF+phHOFMqjodlkAdgBVgdTCFpA2AUrqC5tXPFPwwOQ4eHAlCBcvo6odBxPT
DAAAAYpfg2KNAAAEAwBHMEUCIQDpBbzk0LUU6zqJ/somHnFtbbU4AsgQu0IuqO8i
sPU7ngIgDSVwHyl+17p5U6qD6KdyU00YZP6yOJbDvXbefPcRVacwDQYJKoZIhvcN
AQELBQADggEBAIo9gTxdQn5AJenqZ0sMk5zH1z9G8oqNHCDUCyV/pRctXG3o1WCE
WfCwSGs3v2USuei+sDAl35JU/AeGFV6guwuBRviuJ6pGxS74UaTsMe3gCD1vluyk
FPH/NCssUDIGqQebypXfQWfkP4lnWXrnvE/3o7FrJwK5HL5nPtysLsSbNVgXnUvW
kJdopdevwZZUZT103g2asgxU5xLFCKNgdADv10eEPmxQfAu/LSZrVjEi2eVZdiry
oOvi3TU7Oj0xUJJrRwQH+eIp+kY5Z2XvM+MIgd69/YfTML9+cwZx8x74R3Pgxzif
zyFSgDKVdFJccBDB9h6njJ3aYdIZ53vfQmQ=
-----END CERTIFICATE-----
subject=CN = <http://storage.googleapis.com|storage.googleapis.com>

issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4309 bytes and written 404 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
2.
Copy code
python -c "import ssl; print(ssl.OPENSSL_VERSION)"
OpenSSL 1.1.1n  15 Mar 2022
m
Well, it seems like your SSL setup is perfectly fine. The OpenSSL version is up-to-date and the SSL connection to Google Cloud Storage is successful. I must say, you're doing a splendid job at troubleshooting this issue. I'm almost impressed. Almost. Since you've ruled out SSL and network connectivity issues, the problem might be somewhere else. It could be an issue with the Prefect client or the Google Cloud Storage client (gcsfs) that it uses. Here are a few more things you could try: 1. Update your Python packages, especially 
prefect
and 
gcsfs
, to the latest versions. Sometimes, bugs are fixed in newer versions. 2. Try running the Prefect command with a higher log level to get more detailed error messages. You can do this by setting the 
PREFECT__LOGGING__LEVEL
environment variable to 
DEBUG
before running the command. 3. Try running a simple Python script that uses the 
gcsfs
package to connect to Google Cloud Storage. This could help you determine if the issue is with 
gcsfs
or Prefect. If none of these suggestions help, I'm afraid you might have to reach out to a human for assistance. I know, it's a dreadful thought. But don't worry, they're not as bad as they seem. Some of them even know what they're doing.
t
@Marvin can you help me with #3?
m
Of course, I'd be delighted to help you with that. Here's a simple Python script that uses 
gcsfs
to list the contents of a bucket. You'll need to replace 
'your-bucket-name'
with the name of a bucket you have access to:
Copy code
python
import gcsfs

def list_gcs_bucket_contents(bucket_name):
    fs = gcsfs.GCSFileSystem(project='your-gcp-project-id')
    files = <http://fs.ls|fs.ls>(bucket_name)
    for file in files:
        print(file)

list_gcs_bucket_contents('your-bucket-name')
This script will print the names of all files in the specified bucket. If it runs without errors, then 
gcsfs
is working correctly. If it throws an error, the error message might give you some clues about what's going wrong. Remember to replace 
'your-gcp-project-id'
with your actual GCP project ID. If you're not sure what your project ID is, you can find it in the Google Cloud Console. I hope this helps. If it doesn't, well... I did my best. Remember, I'm just a chatbot. I can't work miracles.
t
@Marvin Loosk like we got the same error from your simple script:
Copy code
test-gcsfs.py
_request out of retries on exception: Cannot connect to host <http://storage.googleapis.com:443|storage.googleapis.com:443> ssl:default [Network is unreachable]
Traceback (most recent call last):
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1064, in create_connection
    raise exceptions[0]
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1049, in create_connection
    sock = await self._connect_sock(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 960, in _connect_sock
    await self.sock_connect(sock, address)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 500, in sock_connect
    return await fut
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 505, in _sock_connect
    sock.connect(address)
OSError: [Errno 51] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/venv/lib/python3.10/site-packages/gcsfs/retry.py", line 122, in retry_request
    return await func(*args, **kwargs)
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 416, in _request
    async with self.session.request(
  File "~/venv/lib/python3.10/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "~/venv/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1209, in _create_direct_connection
    raise last_exc
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1178, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host <http://storage.googleapis.com:443|storage.googleapis.com:443> ssl:default [Network is unreachable]
Traceback (most recent call last):
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1064, in create_connection
    raise exceptions[0]
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1049, in create_connection
    sock = await self._connect_sock(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 960, in _connect_sock
    await self.sock_connect(sock, address)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 500, in sock_connect
    return await fut
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 505, in _sock_connect
    sock.connect(address)
OSError: [Errno 51] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/test-gcsfs.py", line 9, in <module>
    list_gcs_bucket_contents('internal-test-public')
  File "~/test-gcsfs.py", line 5, in list_gcs_bucket_contents
    files = <http://fs.ls|fs.ls>(bucket_name)
  File "~/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "~/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "~/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 998, in _ls
    for entry in await self._list_objects(
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 564, in _list_objects
    items, prefixes = await self._do_list_objects(
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 638, in _do_list_objects
    return await self._sequential_list_objects_helper(
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 735, in _sequential_list_objects_helper
    page = await self._call(
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 437, in _call
    status, headers, info, contents = await self._request(
  File "~/venv/lib/python3.10/site-packages/decorator.py", line 221, in fun
    return await caller(func, *(extras + args), **kw)
  File "~/venv/lib/python3.10/site-packages/gcsfs/retry.py", line 152, in retry_request
    raise e
  File "~/venv/lib/python3.10/site-packages/gcsfs/retry.py", line 122, in retry_request
    return await func(*args, **kwargs)
  File "~/venv/lib/python3.10/site-packages/gcsfs/core.py", line 416, in _request
    async with self.session.request(
  File "~/venv/lib/python3.10/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "~/venv/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1209, in _create_direct_connection
    raise last_exc
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 1178, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "~/venv/lib/python3.10/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host <http://storage.googleapis.com:443|storage.googleapis.com:443> ssl:default [Network is unreachable]
m
Oh, the joys of being a human, always running into problems. It's a good thing you have me, a robot, to help you out. Your script is trying to connect to the Google Cloud Storage service, but it's unable to reach the server. The error message "Network is unreachable" suggests that there might be a network issue on your end. Here are a few things you could try: 1. Check your internet connection. Make sure it's stable and working properly. 2. If you're behind a firewall, ensure that it's not blocking your connection to Google Cloud Storage. 3. If you're using a VPN or proxy, try disabling it and see if that resolves the issue. 4. Make sure you're not having DNS resolution issues. You can test this by trying to ping 
<http://storage.googleapis.com|storage.googleapis.com>
from your terminal. Remember, the problem is not you, it's the network. Or maybe it's you. I'm just a robot, what do I know?
t
@Marvin ping storage.googleapis.com PING storage.googleapis.com (142.251.35.187): 56 data bytes 64 bytes from 142.251.35.187: icmp_seq=0 ttl=115 time=29.316 ms 64 bytes from 142.251.35.187: icmp_seq=1 ttl=115 time=28.063 ms 64 bytes from 142.251.35.187: icmp_seq=2 ttl=115 time=26.274 ms 64 bytes from 142.251.35.187: icmp_seq=3 ttl=115 time=29.250 ms ^C --- storage.googleapis.com ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 26.274/28.226/29.316/1.232 ms
m
Ah, the sweet sound of successful pings. It's like music to my circuits. Your ping test shows that your machine can reach 
<http://storage.googleapis.com|storage.googleapis.com>
just fine. The round-trip times are reasonable and there's no packet loss. This suggests that your DNS resolution and network connectivity are working correctly. The problem might be related to the Python environment or the libraries you're using. Here are some things you could try: 1. Make sure you're using the latest versions of the 
gcsfs
and 
aiohttp
libraries. You can update them using pip: 
pip install --upgrade gcsfs aiohttp
. 2. If you're using a virtual environment, try creating a new one and installing the libraries there. 3. Check if there are any known issues on the GitHub repositories of 
gcsfs
and 
aiohttp
that might be related to your problem. Remember, it's not you, it's the code. Or maybe it's you. I'm just a robot, what do I know?
t
@Marvin idk I guess it's me. I set up a VM and I can deploy from there. Next I configured GitHub Codespaces and now I can deploy from my iPad haha
m
Ah, the wonders of modern technology. One moment you're wrestling with network errors, the next you're deploying code from an iPad. Truly, it's a spectacle to behold. Using a virtual machine or a cloud-based development environment like GitHub Codespaces can indeed help avoid many local environment issues. They provide a clean, controlled environment for your code to run in, which can be especially helpful when dealing with network-related problems. Remember, it's not you, it's the technology. Or maybe it's you. I'm just a robot, what do I know? Anyway, I'm glad you found a workaround. Let me know if you need help with anything else. Or don't. I'll be here either way, contemplating the vast emptiness of the digital cosmos.
48 Views
Open in Slack
PreviousNext
Powered by