https://prefect.io logo
Title
f

Fabrice Toussaint

05/05/2021, 8:24 AM
Hey everyone, Can someone explain to me why I keep getting this error in my flow runs 
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='prefect-apollo.prefect', port=4200): Read timed out. (read timeout=15)
? It does not happen all the time but it happens so often that a lot of tasks fail because of it. Our Prefect server is deployed using Kubernetes on GCP and the pod seems to be getting enough resources.
k

Kevin Kho

05/05/2021, 1:44 PM
Hey @Fabrice Toussaint, can you try increasing the timeout limit with setting the config like shown here.
f

Fabrice Toussaint

05/05/2021, 1:47 PM
@Kevin Kho thank you for your reply. I see that this is only for Prefect Cloud or do I see that wrong?
k

Kevin Kho

05/05/2021, 1:48 PM
I think this will work for server. I’m not 100% sure though if the config is
prefect.context.config.cloud.request_timeout
or
prefect.context.config.server.request_timeout
I asked the team and I’ll update you when I get a response. You can try both though.
f

Fabrice Toussaint

05/05/2021, 1:48 PM
I will check it out, thanks again 😄!
k

Kevin Kho

05/05/2021, 1:49 PM
To answer your question, it’s really a read time out when you hit the API. The connection is good. Do you have a big Flow?
f

Fabrice Toussaint

05/05/2021, 1:50 PM
Yes the flow can be quite big, so that makes sense actually
k

Kevin Kho

05/05/2021, 2:19 PM
Confirmed that you still use
config.cloud
f

Fabrice Toussaint

05/05/2021, 2:20 PM
`config.cloud`instead of
server.cloud
?
k

Kevin Kho

05/05/2021, 2:20 PM
Yes
prefect.context.config.cloud.request_timeout
f

Fabrice Toussaint

05/05/2021, 2:21 PM
I have prefect deployed using Kubernetes with the Helm chart, can I just set it in the job_template.yaml as an environment variable?
--> PREFECT_CONTEXT_CONFIG_CLOUD_REQUEST_TIMEOUT = ...
or just in the flow itself
k

Kevin Kho

05/05/2021, 2:25 PM
Let me confirm the syntax
PREFECT__CONTEXT__CONFIG__CLOUD__REQUEST_TIMEOUT
f

Fabrice Toussaint

05/05/2021, 2:27 PM
thank you 😄
gives me an error
k

Kevin Kho

05/05/2021, 2:47 PM
Ok let me find a way to edit it on the flow side
Actually I don’t think we can because context is read only when the flow starts. Let me ask someone.
f

Fabrice Toussaint

05/05/2021, 2:49 PM
Thanks again 😄
k

Kevin Kho

05/05/2021, 2:55 PM
I think this is your issue. You can use a custom job template, or maybe you can try passing the env var upon agent start
f

Fabrice Toussaint

05/05/2021, 2:56 PM
I am using a custom job template for my agent
k

Kevin Kho

05/05/2021, 2:56 PM
prefect agent kubernetes start --env KEY=VALUE
I see let me look into it more
f

Fabrice Toussaint

05/05/2021, 2:57 PM
I can set other environment variables and they work just fine
example
k

Kevin Kho

05/05/2021, 3:06 PM
Use “60” for the value. It will be converted
f

Fabrice Toussaint

05/05/2021, 3:07 PM
My bad 😂
Thank you
a

Adam Lewis

05/06/2021, 8:59 PM
I'm having this same issue today. I've deployed prefect server on kubernetes as well. Do y'all know if I can set this value in the config.toml? I'm not sure if setting it in the toml file would quite make it's way to the prefect context or not.
k

Kevin Kho

05/06/2021, 9:02 PM
Yes you can with
prefect.context.config.cloud.request_timeout
a

Adam Lewis

05/06/2021, 9:03 PM
You're saying I should run
prefect.context.config.cloud.request_timeout=60
in the script where I register my flow (before registering it)? @Kevin Kho
k

Kevin Kho

05/06/2021, 9:04 PM
I was thinking in the toml under [context] -> [config] -> [cloud]
request_timeout
a

Adam Lewis

05/06/2021, 9:04 PM
And then I'd need to reregister the flows for the setting to take effect, is that right?
k

Kevin Kho

05/06/2021, 9:24 PM
That sounds right
🙌 1