07/28/2023, 8:59 PM
Hello, looking for some help with an issue I encountered while using the
to execute a group of functions (under one task) in parallel. Unfortunately, the flow never completes, and it gets stuck somewhere during the process. I can observe on the Dask dashboard that workers are set up, and work is allocated, but at some point, the process hangs, and the program never terminates. I am facing this issue specifically on Prefect local. I tried running the same code with Dask directly, and surprisingly, it worked without any problems. Through investigation, I narrowed down the issue to a specific function called
. This function doesn't terminate when using Prefect's DaskTaskRunner, but it works fine with normal Dask execution. The
function implementation seems simple, involving regular math computations and converting objects to zarr format for local writing. Each task writes its individual file, and there is nothing complicated happening within
. For reference, here is the implementation of the
function: to_zarr function. I would greatly appreciate any insights or assistance in resolving this issue with the
function while using Prefect's DaskTaskRunner.