Thread
#prefect-server
    t

    Tara

    11 months ago
    Hi, Im running a prefect server on a local server and my flow is using Local dask executor (threads). I occasionally ran into this error where the task runner failed to set state:
    Failed to set task state with error: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='localhost', port=4200): Read timed out. (read timeout=15)"))
    This has been an ongoing issue, anyone has any idea on how to fix this?
    Kevin Kho

    Kevin Kho

    11 months ago
    Hey @Tara, looks like you're running into timeout issues with the API. You can increase the timeout and maybe that will help. Check this
    t

    Tara

    11 months ago
    Thank you @Kevin Kho, I will try increasing the timeout to 30s and hopefully it’s sufficient 🙂
    May I ask what could cause the prefect server to be unavailable for over 15secs ? Since this seems to be a persisting issue.
    Kevin Kho

    Kevin Kho

    11 months ago
    It’s not that it’s unavailable. It’s more of the API requests to it seem to be large and take some time to process.
    t

    Tara

    11 months ago
    To my understanding the API requests from my errors seems to be requests to set state of the task, how can it take so long to process ? please correct me if I’m wrong, i may not have a good understanding of how prefect work yet 😅. Btw I’ve set the request_timeout to 60 and there’s been no issue so far 🙂
    Kevin Kho

    Kevin Kho

    11 months ago
    Is the error happening to you mid run or at the start of the flow run?
    t

    Tara

    11 months ago
    mostly mid run.
    Kevin Kho

    Kevin Kho

    11 months ago
    Ok will ask the team
    This would be the task run state updates are taking too long. How many tasks do you have in your flow and what are the specs of your server? Specifically the API container/pod?
    t

    Tara

    11 months ago
    34 tasks in the flow. 8 x 2GHz CPU, 65GB RAM.
    After setting the request timeout to 60s, I now have errors at the start of a few flow runs (not seeing any mid run error yet).
    ohh, and I’m using prefect 0.15.4 🙂
    Anna Geller

    Anna Geller

    11 months ago
    @Tara are you getting the same errors as before now?
    also, which agent do you use - is it the default Local agent?
    t

    Tara

    11 months ago
    Hi @Anna Geller, so far Im not getting any error as before. Now this is the error I sometimes get at the beginning of the flow. I’m using the default local agent.
    but the error is much less frequent now 🙂
    Kevin Kho

    Kevin Kho

    11 months ago
    Are any of those tasks big mapped tasks? That error seems like the config didn't apply since the read timeout is 15 secs?
    t

    Tara

    11 months ago
    there is no mapped task in my flow. Regarding the timeout, I’ve added this in my .prefect/config.toml and restarted prefect server.
    [cloud]
    request_timeout=60
    At first I thought the new timeout took effect because when I first registered a flow after setting the new timeout, I got a request timeout error (with request timeout=60). The second attempt to register a flow was a success. I’m not sure why I still see timeout=15 when an agent deployed a flow.
    Is there a way to check for request_timeout by querying a Prefect API just to confirm?
    Kevin Kho

    Kevin Kho

    11 months ago
    I believe this will be on the flow run side so the way to confirm is my logging it from context. You can try printing
    prefect.context.cloud.request_timeout
    maybe.
    t

    Tara

    11 months ago
    yupp the request_timeout is 60 by printing
    prefect.context.config.cloud
    .