Hi all, Has anyone here encountered issues when writing to BigQuery from GKE? I'm experiencing intermittent failures with some of my tasks. Specifically, I'm getting a

RetryError

related to a deadline exceeded while making an HTTPS request to BigQuery. Here's the error message:

Finished in state Failed('Task run encountered an exception RetryError: Deadline of 600.0s exceeded while calling target function, last exception: HTTPSConnectionPool(host='<http://bigquery.googleapis.com|bigquery.googleapis.com>', port=443): Max retries exceeded with url: /bigquery/v2/projects/mytables/insertAll?prettyPrint=false (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))')"

Additionally, I'm noticing errors in the GKE logs related to resource constraints:

pod didn't trigger scale-up: 3 Insufficient cpu, 3 Insufficient memory"
(combined from similar events): 0/4 nodes are available: 1 node(s) had untolerated taint {ToBeDeletedByClusterAutoscaler: 1705586686}, 3 Insufficient cpu, 3 Insufficient memory. Preemption: 0/4 nodes are available: 1 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod."

These errors seem to suggest issues with resource allocation and node scaling in the cluster. If anyone has faced and resolved similar problems, your insights would be greatly appreciated!