ash11/23/2022, 11:17 AM
The problem is dask cluster is not shutting until save_data function completes but i expected that as soon as
with Flow("Test flow") as flow: with DaskCluster(n_workers=n_workers) as client: data = extract() processed_data= transform(data) save_data(processed_data)
function completes , the
should happen, Is there any way i can initiate
is done and the
is done as well.
Tim-Oliver11/23/2022, 11:22 AM
ash11/23/2022, 11:23 AM
Tim-Oliver11/23/2022, 11:26 AM
Tim Galvin11/23/2022, 12:24 PM
before exiting? Also, does using the
method on each of the tasks make a difference?