@Marvin how can I use prefect_gcp cloud storage api methods to upload a 2GB .ndjson file? I get

e_2025_03_18_2025/searching_details.ndjson'.
22:35:24.466 | INFO    | Task run 'upload_file_to_gcs-79c' - Failed to upload /mnt/c/Users/user/Documents/c2c/prefect_langgraph_elt/intermediate_outputs/ho-uat-matching/searching_details.ndjson: Timeout of 120.0s exceeded, last exception: HTTPSConnectionPool(host='<http://storage.googleapis.com|storage.googleapis.com>', port=443): Max retries exceeded with url: /upload/storage/v1/b/ho-uat-matching_datalake/o?uploadType=resumable&upload_id=AKDAyIvS8P9L4sr0yi0K0dAxjxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxMCjJK6nEgQ (Caused by SSLError(SSLError(5, '[SYS] unknown error (_ssl.c:2427)')))
22:35:24.469 | INFO    | Task run 'upload_file_t

To upload a large file like a 2GB

.ndjson

to Google Cloud Storage using

prefect_gcp

, you can leverage the

GcsBucket

class, which supports uploading and downloading entire directories. However, for large files, you might encounter timeout issues like the one you've experienced. Here are some steps and considerations to handle this: 1. Increase Timeout: If possible, try increasing the timeout setting. However, be cautious as this might not always be the best solution for very large files. 2. Chunked Uploads: Consider breaking the file into smaller chunks and uploading each chunk separately. This can help bypass timeout issues and is generally more reliable for large files. 3. Retry Mechanism: Implement a retry mechanism in your task to handle transient network issues or timeouts. 4. Check Network: Ensure that there are no network issues or restrictions that might be causing the timeout. 5. Use

GcsBucket

: Refer to the Prefect GCP documentation for specific methods within

GcsBucket

that might support your use case. For your specific error, it seems there might be an SSL error. Ensure that your SSL certificates are up to date and correctly configured. If you continue to face issues, you might want to consult the Google Cloud documentation for additional strategies on handling large file uploads. For more details on

prefect_gcp

, you can explore the module offerings within Prefect GCP.