Has anyone run into this error? Only happens when we use the Dask executor, always at the same spot ...

Erik Amundson

10/03/2023, 8:29 PM

Has anyone run into this error? Only happens when we use the Dask executor, always at the same spot (right before the first mapped task). Also only happens in kubernetes, not when running locally.

Copy code

Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Client error '408 Request Timeout' for url '<https://api.prefect.cloud/api/accounts/><xxx>/workspaces/<xxx>/task_runs/'

Emil Christensen

10/03/2023, 9:07 PM

Hey @Erik Amundson 👋 we’re actively investigating the underlying reason for the 408s, but as a mitigation you can set

Copy code

PREFECT_CLIENT_RETRY_EXTRA_CODES=408

which will retry in case of 408.

Emil Christensen

10/03/2023, 9:09 PM

@Erik Amundson If you’re up for it, could you try seeing if allocating more CPU in k8s resolves the issue? We think the error may have to do with resource contention.

Erik Amundson

10/03/2023, 9:12 PM

It's on a 16 vCPU node running a local dask cluster, but I can try manually setting the number of workers to lower so they each have more.

🙏 1

Andrew Lawlor

12/14/2023, 2:41 PM

I'm running into this issue too. Were you able to fix it?

4 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.