ash11/22/2022, 2:51 PM
The problem is dask cluster is not shutting until save_data function completes but i expected that as soon as
with Flow("Test flow") as flow: with DaskCluster(n_workers=n_workers) as client: data = extract() processed_data= transform(data) save_data(processed_data)
function completes , the
should happen, Is there any way i can initiate
is done and the
is done as well.
Bianca Hoch11/23/2022, 8:56 PM
is completed. I'll investigate a bit to see if there is a way to specify that. In the meantime, here are some resources that may be helpful. • How can I configure my flow to run with Dask?
ash11/24/2022, 6:55 AM