Hi. I keeping getting flows failing with `HTTPSCon...
# ask-communityt
Thomas Hoeck
09/08/2020, 7:17 AMHi. I keeping getting flows failing with
HTTPSConnectionPool(host='10.0.0.1', port=443): Read timed out. (read timeout=None)Thomas Hoeck
09/08/2020, 7:55 AMMight be related to this https://github.com/kubernetes-client/python/issues/990
Thomas Hoeck
09/08/2020, 8:01 AMI might be related to the job manager becasuse according to this it might be
self.batch_client.list_namespaced_jobj
Jenny
09/08/2020, 2:00 PMHi @Thomas Hoeck - Thanks for the question. I'll check with the wider time to see if they have any insight. Hopefully others can chime in here if they've see anything similar.
t
Thomas Hoeck
09/08/2020, 2:11 PMI have created an Issue on https://github.com/PrefectHQ/prefect/issues/3278
Thomas Hoeck
09/08/2020, 2:11 PMThank you @Jenny 🙂
👍 1
j
Jim Crist-Harif
09/08/2020, 2:14 PMHi Thomas, the prefect agent doesn't start a watch, so that's unlikely to be it, but it may be related. It's curious that you're seeing failed flow runs from this, errors at the k8s level shouldn't (necessarily) bubble up to be flow run failures.
You say you're using the k8s agent, what prefect
Environmentt
Thomas Hoeck
09/08/2020, 2:16 PM@Jim Crist-Harif I'm using LocalEnvironment. Will quickly see what logs I can dig out.
I'm running prefect 0.13.5.
Thomas Hoeck
09/08/2020, 2:22 PMThis is from the agents logs:
j
Jim Crist-Harif
09/08/2020, 2:27 PMIs the agent running prefect 0.13.5 as well?
Jim Crist-Harif
09/08/2020, 2:27 PMThanks, those logs are helpful
t
Thomas Hoeck
09/08/2020, 2:28 PM@Jim Crist-Harif Yes agent is running 0.13.5 as well.
Thomas Hoeck
09/08/2020, 2:34 PM@Jim Crist-Harif I can also add that it looks like this error either causes or is caused by the containers never starting i.e. the containers for the failed flows newer started.
j
Jim Crist-Harif
09/08/2020, 2:34 PMOk, so the error here appears to be a timeout on POST to the k8s api. We'll need to redo the error handling here to support retries in the agent. I see you've opened an issue for this, I'll comment there, and we can continue the work in the repo.
t
Thomas Hoeck
09/08/2020, 2:40 PM@Jim Crist-Harif Makes sense because if I restart the run it works as expected.
2 Views