Has anyone run into this error? Only happens when ...
# ask-community
e
Has anyone run into this error? Only happens when we use the Dask executor, always at the same spot (right before the first mapped task). Also only happens in kubernetes, not when running locally.
Copy code
Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Client error '408 Request Timeout' for url '<https://api.prefect.cloud/api/accounts/><xxx>/workspaces/<xxx>/task_runs/'
e
Hey @Erik Amundson 👋 we’re actively investigating the underlying reason for the 408s, but as a mitigation you can set
Copy code
PREFECT_CLIENT_RETRY_EXTRA_CODES=408
which will retry in case of 408.
@Erik Amundson If you’re up for it, could you try seeing if allocating more CPU in k8s resolves the issue? We think the error may have to do with resource contention.
e
It's on a 16 vCPU node running a local dask cluster, but I can try manually setting the number of workers to lower so they each have more.
🙏 1
a
I'm running into this issue too. Were you able to fix it?