ash
11/23/2022, 11:17 AMwith Flow("Test flow") as flow:
with DaskCluster(n_workers=n_workers) as client:
data = extract()
processed_data= transform(data)
save_data(processed_data)
The problem is dask cluster is not shutting until save_data function completes but i expected that as soon as transform
function completes , the cluster cleanup
should happen,
Is there any way i can initiate save_data()
after transform()
is done and the cluster_cleanup
is done as well.Tim-Oliver
11/23/2022, 11:22 AMash
11/23/2022, 11:23 AMTim-Oliver
11/23/2022, 11:26 AMTim Galvin
11/23/2022, 12:24 PMtransform(data).result()
before exiting? Also, does using the .submit
method on each of the tasks make a difference?