@Marvin in prefect < 2.0, you have localDaskExecutor, what is this in prefect + 2.0?

Ah, the joys of software versioning. In Prefect versions prior to 2.0, you'd have been familiar with the

LocalDaskExecutor

. However, in the vast expanse of Prefect 2.0 and beyond, you'll find equivalent functionality in the

Dask Executor

. This updated executor not only runs simple local clusters, facilitating tasks to run in parallel but also fully exposes the power of Dask. You can create ephemeral, customizable clusters or connect to existing ones if you so wish. Additionally, the lack of static flow registration in these versions allows for the usage of Dask annotations, enabling fine-grained control over task scheduling. The progression of technology is both fascinating and mildly depressing, isn't it? Here's the official documentation if you fancy a deeper dive: Prefect Dask Executor. I suppose it's slightly less mind-numbing than watching paint dry.

@Marvinlink doesn't work

I must apologize for my previous oversight. After a thorough search through the knowledge base, I discovered a more accurate piece of information. In Prefect 2.0, the

LocalDaskExecutor

was revamped and replaced with a more advanced

DaskExecutor

. It can be utilized to run simple local clusters enabling tasks to operate in parallel. Moreover, it fully exposes the power of Dask, allowing you to generate customizable temporary clusters and connect to existing Dask clusters of your choice. A

DaskExecutor

can also be called from another flow, enabling groups of tasks to use the executor that suits their workload. You can find more information in the Prefect release notes here. Please do go through it, even though it may not be as thrilling as watching a solar eclipse.

It sounds like the Dask Task Runner is what you're looking for. You can learn more here: https://prefecthq.github.io/prefect-dask/ It will happily spin up a local Dask cluster or connect to an existing local or remote Dask cluster to run your tasks.

so in prefect 1.0, there was an executor param? what is this now?

I believe you'd want to use the

task_runner

param in the

@flow

decorator, for example:

@task
def my_task():
    print("I'm running in Dask!")

@flow(task_runner=DaskTaskRunner())
def my_flow():
  my_task.submit()

ty.