Hi Team,
We came across this error twice in a mon...
# ask-communitys
Saurabh Indoria
10/25/2022, 3:04 AMHi Team,
We came across this error twice in a month, and it makes us worry about the reliability of Prefect. Since Prefect is the backbone of our infrastructure, everything stops working when this happens. The retries and everything simply fails and all our flows fail until this issue persists.
Our deployment: Prefect Cloud (1.0) + GCP Kubernetes + LocalDaskExecutor
What can be done to safeguard against this? Is there a reliable way to retry and not fail?
Task Run for Reference.
CC: @Christina Lopez @Yash Joshi
Copy code
[20 Oct 2022 12:22pm]: Error during execution of task: SSLError(MaxRetryError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))"))m
Mason Menges
10/25/2022, 3:18 PMHey @Saurabh Indoria This error can come up for a variety of reasons, in this case the task run appears to have failed due to a heartbeat error and then the prefect api likely reached the maximum attempts to restart the task run resulting in this error
Error during execution of task: SSLError(MaxRetryError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443)s
Saurabh Indoria
10/26/2022, 3:43 AMThanks @Mason Menges 🙌
💙 1
Saurabh Indoria
11/11/2022, 8:44 AM@Mason Menges I saw this error again on this flow run and this time the heartbeat issue came after a long time following this error. I assume due the Prefect API not reachable, obviously the heartbeats won't be reported. This is definitely not an issue about the pod being dropped because there were multiple tasks which reported this issue before the heartbeat issue coming up.
Please let me know if my understanding is incorrect here..
CC: @Christina Lopez
👀 1
c
Christina Lopez
11/11/2022, 3:07 PMFlagging for @Kristen Denk as well since she’s helping y’all with your evaluation.
👀 1
🙌 1
k
Kristen Denk
11/15/2022, 2:55 PM@Taylor Curran
Kristen Denk
11/15/2022, 3:04 PM@George Coyne
g
George Coyne
11/15/2022, 6:11 PMThis looks like a TLS error
George Coyne
11/15/2022, 6:16 PMYou can check that with something like
Copy code
from urllib.request import urlopen
urlopen('<https://www.howsmyssl.com/a/check').read(>)George Coyne
11/15/2022, 6:16 PMYour python should be running TLS 1.2
George Coyne
11/15/2022, 6:16 PMWe also see this with proxy config
George Coyne
11/15/2022, 6:17 PMBecause on our side there are no API stability or SSL issues
k
Kristen Denk
11/15/2022, 7:55 PM@Saurabh Indoria @Yash Joshi
🙏 2
s
Saurabh Indoria
11/16/2022, 1:53 AMThanks for that insight, but I wonder why this would occur once in a thousand runs. If our TLS config is bad, the flows should never run, right? @George Coyne
a
Anna Geller
11/21/2022, 3:33 PM
correct, perhaps some transient issue that you could solve with flow-level retries? that's what retries are for 🙌