https://prefect.io
Join Slack
Hi! Has anyone tried to use the <dask.distributed performance_report> with a Prefect DaskExecutor ? ...
# ask-community
m
Hi! Has anyone tried to use the dask.distributed performance_report with a Prefect DaskExecutor ? Would this be a good place to inject it? https://github.com/PrefectHQ/prefect/blob/b45029d00579ef6ad9a9479e5eb03b3e21ee31b7/src/prefect/executors/dask.py#L232 Thanks
k
Hi @Michal Mucha, I haven’t used it myself but I think the easier way to do it would be to do something like this:
Copy code
from dask.distributed import worker_client


@task
def calling_compute_in_a_task(filepath):
    with performance_report(filename="dask-report.html"):
        with worker_client():
            df = dd.read_csv(filepath)
            return df.describe().compute()
This is if you’re using Dask commands with Prefect. If you’re not using Dask commands, and instead using Dask through Prefect, then I guess you might need to inject it there.
m
Thanks Kevin. Yeah I wanted to get the profile flamegraph (that Dask makes across a cluster) on a Dask cluster spawned to execute a Prefect flow. For the entire flow
rather than to profile a Dask workload where the compute is called in a task
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by