https://prefect.io
Join Slack
Hi folks. Does anyone know how I generate a `Dask Performance Report` of a Prefect Flow that is usin...
# ask-community
w
Hi folks. Does anyone know how I generate a 
Dask Performance Report
of a Prefect Flow that is using 
DaskExecutor
i.e. https://distributed.dask.org/en/latest/api.html#distributed.performance_report It looks like the 
dask.distributed.performance_report()
needs to run inside the context of the temporary Dask Cluster that Prefect spins up, but I don’t see how to get into those internals.
k
I was curious about this myself. I think you need a task like this:
Copy code
@task(checkpoint=False)
def using_checkpoint_false(filepath):
    with worker_client():
        with performance_report(filename="dask-report.html"):
            return dd.read_csv(filepath)
You also might be able to try:
Copy code
with Flow("example") as flow:
    with performance_report(...):
        some_task()
        other_task()
and this might run if the flow is stored as a script
w
Cool, thanks, I will try this
b
I’m also interested in getting this performance report -do let us know if you are able to get it to work
w
Finally getting to this this afternoon; I'll report back.
Hmm. • The first suggestion at least runs, but it only gives the performance report for the single task the performance report context manager wraps • The second approach just throws an error. We’re using module storage.
k
Yikes in that case it will be pretty hard to add it because 
executor.submit()
is agnostic to the full workflow already. If you have a mapped task with 5 elements, 
executor.submit()
calls them one by one. I am not sure there is a good place to inject this then. Will ask the team for ideas.
w
Thanks!
k
b
👍
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by