https://prefect.io
Join Slack
Hello prefecteers! I'm having some trouble with what appears to be a deadlock. I've gotten a minimum...
# ask-community
m
Hello prefecteers! I'm having some trouble with what appears to be a deadlock. I've gotten a minimum reproducible example that may or may not be the cause for my issue, which centers around tasks that submit additional work to the dask client which serves as the executor. I am using prefect core with the dask executor. Here is the example:
Untitled
Obviously the example is quite stilted, but I got here due to a hang when using sklearn's joblib_parallel with the dask back-end during model selection. I wouldn't think this would be some crazy novel thing, but I also can't find evidence of anybody running into the same problem.
z
I think you need to run your dask client code in a 
with worker_client
block. See https://distributed.dask.org/en/latest/task-launch.html#connection-with-context-manager
this can deadlock the scheduler if too many tasks request jobs at once. Each task does not communicate to the scheduler that they are waiting on results and are free to compute other tasks. This can deadlock the cluster if every scheduling slot is running a task and they all request more tasks.
m
That has absolutely done it for the small testcase, thanks a ton Michael. I'll take it to the non-trivial case next!
z
@Marvin archive "Encountering deadlock using 
dask.Client
in task"
m
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by