https://prefect.io logo
s

Soham

07/28/2023, 8:56 PM
@Marvin: Hello, looking for some help with an issue I encountered while using the DaskTaskRunner to execute a group of functions (under one task) in parallel. Unfortunately, the flow never completes, and it gets stuck somewhere during the process. I can observe on the Dask dashboard that workers are set up, and work is allocated, but at some point, the process hangs, and the program never terminates. I am facing this issue specifically on Prefect local. I tried running the same code with Dask directly, and surprisingly, it worked without any problems. Through investigation, I narrowed down the issue to a specific function called
to_zarr
. This function doesn't terminate when using Prefect's DaskTaskRunner, but it works fine with normal Dask execution. The
to_zarr
function implementation seems simple, involving regular math computations and converting objects to zarr format for local writing. Each task writes its individual file, and there is nothing complicated happening within
to_zarr
. For reference, here is the implementation of the
to_zarr
function: to_zarr function. I would greatly appreciate any insights or assistance in resolving this issue with the
to_zarr
function while using Prefect's DaskTaskRunner.