c

    Chris Eberly

    2 years ago
    last question, and apologies if this has been answered. if folks have an existing Dask-ified task but want to use other parts of Prefect with the DaskExecutor, is it recommended to rewrite the existing task using just prefect or simply run the old task separately
    Chris White

    Chris White

    2 years ago
    Hey Chris! No worries; this is an excellent question; Prefect tasks add a lot of baggage to the Dask scheduler, so I wouldn’t recommend replacing the Dask API with the Prefect API wholesale. The only consideration you’ll need to make with your task-dask logic is that the task will be executed on a worker, and some packages have trouble with this as they assume they are being executed in a process external to the dask cluster
    c

    Chris Eberly

    2 years ago
    thanks!