https://prefect.io logo
Title
t

Tara

10/21/2021, 11:51 PM
Hi, Im running a prefect server on a local server and my flow is using Local dask executor (threads). I occasionally ran into this error where the task runner failed to set state:
Failed to set task state with error: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='localhost', port=4200): Read timed out. (read timeout=15)"))
This has been an ongoing issue, anyone has any idea on how to fix this?
k

Kevin Kho

10/21/2021, 11:53 PM
Hey @Tara, looks like you're running into timeout issues with the API. You can increase the timeout and maybe that will help. Check this
t

Tara

10/21/2021, 11:57 PM
Thank you @Kevin Kho, I will try increasing the timeout to 30s and hopefully it’s sufficient 🙂
May I ask what could cause the prefect server to be unavailable for over 15secs ? Since this seems to be a persisting issue.
k

Kevin Kho

10/22/2021, 12:37 AM
It’s not that it’s unavailable. It’s more of the API requests to it seem to be large and take some time to process.
t

Tara

10/22/2021, 1:14 AM
To my understanding the API requests from my errors seems to be requests to set state of the task, how can it take so long to process ? please correct me if I’m wrong, i may not have a good understanding of how prefect work yet 😅. Btw I’ve set the request_timeout to 60 and there’s been no issue so far 🙂
k

Kevin Kho

10/22/2021, 1:16 AM
Is the error happening to you mid run or at the start of the flow run?
t

Tara

10/22/2021, 1:31 AM
mostly mid run.
k

Kevin Kho

10/22/2021, 2:12 AM
Ok will ask the team
🙏 1
This would be the task run state updates are taking too long. How many tasks do you have in your flow and what are the specs of your server? Specifically the API container/pod?
t

Tara

10/22/2021, 7:42 AM
34 tasks in the flow. 8 x 2GHz CPU, 65GB RAM.
After setting the request timeout to 60s, I now have errors at the start of a few flow runs (not seeing any mid run error yet).
ohh, and I’m using prefect 0.15.4 🙂
a

Anna Geller

10/22/2021, 9:13 AM
@Tara are you getting the same errors as before now?
also, which agent do you use - is it the default Local agent?
t

Tara

10/22/2021, 11:58 PM
Hi @Anna Geller, so far Im not getting any error as before. Now this is the error I sometimes get at the beginning of the flow. I’m using the default local agent.
but the error is much less frequent now 🙂
k

Kevin Kho

10/23/2021, 5:08 PM
Are any of those tasks big mapped tasks? That error seems like the config didn't apply since the read timeout is 15 secs?
:upvote: 1
t

Tara

10/24/2021, 11:49 PM
there is no mapped task in my flow. Regarding the timeout, I’ve added this in my .prefect/config.toml and restarted prefect server.
[cloud]
request_timeout=60
At first I thought the new timeout took effect because when I first registered a flow after setting the new timeout, I got a request timeout error (with request timeout=60). The second attempt to register a flow was a success. I’m not sure why I still see timeout=15 when an agent deployed a flow.
Is there a way to check for request_timeout by querying a Prefect API just to confirm?
k

Kevin Kho

10/25/2021, 12:21 AM
I believe this will be on the flow run side so the way to confirm is my logging it from context. You can try printing
prefect.context.cloud.request_timeout
maybe.
t

Tara

10/25/2021, 10:37 PM
yupp the request_timeout is 60 by printing
prefect.context.config.cloud
.