Hi, Im running a prefect server on a local server and my flow is using Local dask executor (threads). I occasionally ran into this error where the task runner failed to set state:

Failed to set task state with error: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='localhost', port=4200): Read timed out. (read timeout=15)"))

This has been an ongoing issue, anyone has any idea on how to fix this?

Hey @Tara, looks like you're running into timeout issues with the API. You can increase the timeout and maybe that will help. Check this

Thank you @Kevin Kho, I will try increasing the timeout to 30s and hopefully it’s sufficient 🙂

It’s not that it’s unavailable. It’s more of the API requests to it seem to be large and take some time to process.

To my understanding the API requests from my errors seems to be requests to set state of the task, how can it take so long to process ? please correct me if I’m wrong, i may not have a good understanding of how prefect work yet 😅. Btw I’ve set the request_timeout to 60 and there’s been no issue so far 🙂

Is the error happening to you mid run or at the start of the flow run?

mostly mid run.

Ok will ask the team

34 tasks in the flow. 8 x 2GHz CPU, 65GB RAM.

@Tara are you getting the same errors as before now?

Hi @Anna Geller, so far Im not getting any error as before. Now this is the error I sometimes get at the beginning of the flow. I’m using the default local agent.

Are any of those tasks big mapped tasks? That error seems like the config didn't apply since the read timeout is 15 secs?

there is no mapped task in my flow. Regarding the timeout, I’ve added this in my .prefect/config.toml and restarted prefect server.

[cloud]
request_timeout=60

At first I thought the new timeout took effect because when I first registered a flow after setting the new timeout, I got a request timeout error (with request timeout=60). The second attempt to register a flow was a success. I’m not sure why I still see timeout=15 when an agent deployed a flow.

I believe this will be on the flow run side so the way to confirm is my logging it from context. You can try printing

prefect.context.cloud.request_timeout

maybe.

yupp the request_timeout is 60 by printing

prefect.context.config.cloud

.