https://prefect.io logo
Title
m

Matt Alhonte

06/06/2022, 11:01 PM
What's a good way to try multiple model types with the meta-estimators (like
RandomizedSearchCV
)in
dask-ml
?
map
?
k

Kevin Kho

06/06/2022, 11:06 PM
Very naive about this, but doesn’t one meta estimator already use all available workers so you run into resource contention?
Map feels like there’s two levels of abstraction if you know what I mean. It seems off because it should either be
map
or the
dask-ml
right?
m

Matt Alhonte

06/06/2022, 11:09 PM
Maybe! I dunno how it worker under the hood 😅 . Currently just using multiple Prefect tasks, which tbh is fine but I was wondering if I could avoid multiple loads of the data (and it's big enough that passing it through the Scheduler causes problems)