Hi folks Does anyone know how I generate a `Dask Performance Prefect Community #ask-community

Hi folks. Does anyone know how I generate a `Dask ...

Wilson Bilkovich

10/05/2021, 1:30 PM

Hi folks. Does anyone know how I generate a

Dask Performance Report

of a Prefect Flow that is using

DaskExecutor

i.e. https://distributed.dask.org/en/latest/api.html#distributed.performance_report It looks like the

dask.distributed.performance_report()

needs to run inside the context of the temporary Dask Cluster that Prefect spins up, but I don’t see how to get into those internals.

Kevin Kho

10/05/2021, 1:52 PM

I was curious about this myself. I think you need a task like this:

Copy code

@task(checkpoint=False)
def using_checkpoint_false(filepath):
    with worker_client():
        with performance_report(filename="dask-report.html"):
            return dd.read_csv(filepath)

Kevin Kho

10/05/2021, 1:55 PM

You also might be able to try:

Copy code

with Flow("example") as flow:
    with performance_report(...):
        some_task()
        other_task()

and this might run if the flow is stored as a script

Wilson Bilkovich

10/05/2021, 1:58 PM

Cool, thanks, I will try this

Bob Colner

10/05/2021, 2:51 PM

I’m also interested in getting this performance report -do let us know if you are able to get it to work

Wilson Bilkovich

10/05/2021, 4:54 PM

Finally getting to this this afternoon; I'll report back.

Wilson Bilkovich

10/06/2021, 12:52 PM

Hmm. • The first suggestion at least runs, but it only gives the performance report for the single task the performance report context manager wraps • The second approach just throws an error. We’re using module storage.

Kevin Kho

10/06/2021, 2:39 PM

Yikes in that case it will be pretty hard to add it because