Hi Team We came across this error twice in a month and it ma

Question

Hi Team We came across this error twice in a month and it makes us worry about the reliability of Prefect Since Prefect is the backbone of our infrastructure everything stops working when this happens The retries and everything simply fails and all our flows fail until this issue persists Our deployment Prefect Cloud 1 0 + GCP Kubernetes + LocalDaskExecutor What can be done to safeguard against this Is there a reliable way to retry and not fail for Reference CC < Christina Lopez> < Yash Joshi> ``` 20 Oct 2022 12 22pm Error during execution of task SSLError MaxRetryError HTTPSConnectionPool host= port=443 Max retries exceeded with url Caused by SSLError SSLEOFError 8 EOF occurred in violation of protocol ssl c 1131 ```

Mason Menges · Answer

Hey < Saurabh Indoria> This error can come up for a variety of reasons in this case the task run appears to have failed due to a heartbeat error and then the prefect api likely reached the maximum attempts to restart the task run resulting in this error `Error during execution of task SSLError MaxRetryError HTTPSConnectionPool host= port=443 ` this discourse article describes part of the issue here and offers some suggestions to alleviate the issue

Saurabh Indoria · Answer

Thanks < Mason Menges> raised hands

Saurabh Indoria · Answer

< Mason Menges> I saw this error again on flow run and this time the heartbeat issue came after a long time following this error I assume due the Prefect API not reachable obviously the heartbeats won t be reported This is definitely not an issue about the pod being dropped because there were multiple tasks which reported this issue before the heartbeat issue coming up Please let me know if my understanding is incorrect here CC < Christina Lopez>

Christina Lopez · Answer

Flagging for < Kristen Denk> as well since she s helping y all with your evaluation

Kristen Denk · Answer

< Taylor Curran>

Kristen Denk · Answer

< George Coyne>

George Coyne · Answer

This looks like a TLS error

George Coyne · Answer

You can check that with something like ```from urllib request import urlopen urlopen <https www howsmyssl com a check read > ```

George Coyne · Answer

Your python should be running TLS 1 2

George Coyne · Answer

We also see this with proxy config

George Coyne · Answer

Because on our side there are no API stability or SSL issues

Kristen Denk · Answer

< Saurabh Indoria> < Yash Joshi>

Saurabh Indoria · Answer

Thanks for that insight but I wonder why this would occur once in a thousand runs If our TLS config is bad the flows should never run right < George Coyne>

Anna Geller · Answer

correct perhaps some transient issue that you could solve with flow level retries that s what retries are for raised hands