This might not be the proper place to ask, but can anyone help me understand why I’m seeing so much better performance with joblib than prefect/dask? Obviously dask and joblib aren’t exactly apples-to-apples but MAN joblib blows it out of the water in terms of speed for parallel computing on a single machine. A cursory internet search for “joblib vs dask” didn’t yield any satisfactory answer…

Hi @Jake Schmidt, we did some testing with joblib (via sklearn) and Dask quite a while ago and have since moved entirely to running Prefect on a Dask cluster with very good results. In our case, our performance is dominated by ML training & testing time so distributing the work across a large cluster is very beneficial for us. Can you say a little more about the configuration you're testing and how you're comparing? (Also, are you doing something like this with joblib? https://joblib.readthedocs.io/en/latest/auto_examples/parallel/distributed_backend_simple.html)