@Marvin Can I use @dask.delayed with @task ?

It appears you can combine

@dask.delayed

with

@task

when using Prefect, especially within a flow that leverages the

DaskTaskRunner

. Below are some examples and discussions I found relevant to your question: 1. In a documented example, tasks are defined using

@task

and make use of Dask dataframe operations within a Prefect flow that uses the

DaskTaskRunner

. This suggests that Dask and Prefect can be integrated smoothly.

python
   import dask.distributed
   from prefect import flow, task
   from prefect_dask import DaskTaskRunner, get_dask_client

   # ... (task definitions here)
   
   @flow(task_runner=DaskTaskRunner(address=client.scheduler.address))
   def dask_pipeline():
       # ... (flow code here)
       pass

Source: prefect-dask documentation 2. An issue on GitHub discusses recursively calling a function with

dask.delayed

in a Prefect flow and trying to run each task in its own Dask worker pod for parallelization. The user employs Prefect's

@task

and

.submit

methods along with Dask. Source: GitHub Issue 3. Another GitHub Issue indicates that functions must be wrapped with

@task

for them to be submitted in parallel to Dask when using

DaskTaskRunner

. Source: GitHub Issue 4. There is also a mention of image processing pipelines failing due to memory issues when running tasks with

DaskExecutor

. This highlights the need to manage resources efficiently when using Dask and Prefect together. Source: GitHub Issue In summary, it is possible to use

@dask.delayed

with

@task

in a Prefect flow, particularly when using the

DaskTaskRunner

. However, you should take care to manage resources and understand the orchestration behavior to ensure your tasks are executed as intended.