For example what I'm trying to say:

some_xarray_dataset.to_zarr(store, mode='w', consolidated=True)

the plain Python statement above, when using Dask without Prefect, dask will automatically pick up the work to perform in parallel with multiple processes in all threads. But with Prefect, the

task_runner

(

DaskTaskRunner

) won't able to. So I have to wrap it in a task function that looks like:

@task
def xr_to_zarr(dataset):
  dataset.to_zarr(store, mode='w', consolidated=True)

this way, the

xr_to_zarr

task will be added to Dask graph, however, it will be executed in a single CPU thread, although it should be many many tasks of

.to_zarr()

could be performed in all threads Another attempt I tried:

@dask.delayed
def xr_to_zarr(dataset):
  z = dataset.to_zarr(store, mode='w', consolidated=True, compute=False)
  return z

then in the

flow

I'd call it like

z = xr_to_zarr(dataset)

, this returns a Dask delayed object, then I run

z.compute()

, it eventually runs, I can see the worker's memory filled up, but nothing showing up in Task Stream and Progress lists from the Dask dashboard created by

DaskTaskRunner

, so I wonder if this is a bug or

z.compute()

running somewhere else?

you can also use Dask client directly within Prefect 2.0 - you can run any Python there, including Dask code

Can you please provide me a dummy example for the proper way to call Dask client with Prefetch 2.0? That would work I assume, thanks a lot.

works very nice, I didn't know that Dask client can be called directly, not through

DaskTaskRunner

thank you @Anna Geller