Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

simple_linreg_flow_dask.html

Hello I’m trying a simple flow of doing linear regression in batches. The flow works when doing it sequentially but when I try with Dask backend it causes memory problems. What is confusing is that there is ample memory per worker. Can someone help me identify the problem?

The Dask Dashboard looks normal except for more memory consumption than expected (~12GB). But when the task `fit_create_model` finishes it loops around and runs it again infinitely, indicating that something failed