https://prefect.io logo
Join Slack
Powered by
Hello Prefect community, I'm getting a strange `Cl...
# ask-community
c
Hello Prefect community, I'm getting a strange 
ClientError
when using Prefect Cloud for a mapped tasks. The inputs to each task are identical, but some mapped tasks are successful while a few raise 
DeprecationWarning: Using 'method_whitelist' with Retry is deprecated and will be removed in v2.0. Use 'allowed_methods' instead
(see screenshot for trace). Has anybody experienced something similar? I'm using urllib3 v1.26.7 and prefect v0.15.9
a
@Chris L. at first glance, it looks like some version mismatch (I may be wrong). So you registered your flow in 0.15.9 environment with urllib3 v1.26.7. Are the same package versions used by your agent or Dask workers? Can you share which agent type and executor do you use in this flow?
c
I'm using a KubernetesAgent and local dask executor. I've specified a custom Docker image in the KubernetesRunConfig. I register my flows via a GH Actions workflow that rebuilds a conda environment each time, so registration and execution environment mismatch should not be a problem? I'll need to check that nobody accidentally "bypassed" the CI workflow and registered flows directly from their laptop. Will try to rerun the full CI workflow again!
a
I cross-checked the urllib version and your version is fine https://github.com/PrefectHQ/prefect/blob/master/requirements.txt#L25
c
Would K8 job resource requests and limits have anything to do with this issue?
a
it could have something to do with resources, but I would expect that the reason for the Retry of a GraphQL API call itself was some parallel execution and concurrency issue because the 
set_task_run_states
handles concurrency logic. But it doesn’t explain the urllib error for sure. Did the reregistering via CI workflow fix it? It actually could fix the issue
c
Hi I'm afraid I'm still getting the same issue after running the CI pipeline. I've also massively increased the K8 node size...
I don't get this issue when I run the flow locally (i.e. w/o prefect backend)
But every 1/20 local runs, I do get this error https://github.com/conda/conda/issues/9589 and the flow stops running, but this error is not picked up by the Prefect flow runner
a
Could you try urllib3==1.25.11? From this thread: “The thing is that 
method_whitelist
 was deprecated and removed from 
urllib3==1.26.0
 onwards, as stated in the release changelog. The solution in my case was to set the 
urllib
 version to a previous one, adding 
urllib3==1.25.11
 to my 
requirements.txt
”
somehow the mapped task seems to be using an older Prefect version that needs lower urllib3 version? maybe the flow image or the image set on KubernetesRun uses a different Prefect and urllib versions?
c
This could be it. I recently upgraded prefect and set up GitHub Actions Docker build cache. Maybe something weird is going on
Thanks a lot Anna! I'll use these hints and try to get unstuck
🙌 1
Hello @Anna Geller, have no idea what caused the bug. But I refactored my code to avoid CPU and RAM "spikes" and everything seems to be working smoothly. Probably some hardware-software Dask black magic. I didn't have to change any of the dependencies...
Quick update: I've tested the flow using 
prefect flow run --execute
agentless flow run and I am getting the same "method whitelist" error as above. I ran this immediately after registering the flow directly from my laptop inside the same conda environment as the local flow run execution environment. I'm wondering if there is something on the Prefect Cloud side that might be causing this error?
A bit of context: BLP is a logistic regression model. I'm using Prefect mapped tasks to parallelize bootstrapped estimation. So each mapped task run has identical parameters. The number of mapped tasks is determined by the parameter 
n_bootstraps
. I just tested a few values of 
n_bootstraps
. The 
method_whitelist
error only occurs when I use Prefect Cloud with 
n_bootstraps
<= 3. This is super weird because I've used Prefect mapped tasks with over 5000 mapped tasks before with no issue...
a
@Chris L. Thanks for analyzing the situation so thoroughly! I talked to the engineering team yesterday about it and Michael pushed a fix for it: https://github.com/PrefectHQ/prefect/commit/a02b554b2f5d7a985401e71f08c4d203404b79fa It’s not in the official release, but if you install from master, the error should be gone.
10 Views
Open in Slack
PreviousNext
Powered by