Robert Banick
05/04/2023, 11:13 PMon_crashed
parameter just released (!). Is is possible to pass in additional arguments to function so invoked, using objects generated during the course of a flow? Like so:
def cleanup(path_name: str):
shutil.rmtree(path_name)
def crash_hook(path_name, flow, flow_run, state):
cleanup(path_name)
@flow(on_crashed=[crash_hook])
def my_flow():
# stuff
path_name = other_function()
My use case is wiping very large amounts of data downloaded for temporary processing as a part of my ETLs. Leaving this lying around on temporary storage is expensive and best avoided. The path name for the directory to wipe depends on the ETL so invoked and so a static function won’t cut it; I need to vary the path I apply shutil.rmtree
to.
Thanks for pushing this awesome feature, it directly responds to a problem I was just about to address. Very cool!Abhinav Chordia
05/05/2023, 2:29 AMjustabill
05/05/2023, 12:10 PMpath_name
as a string block so it can be retrieved.Robert Banick
05/05/2023, 3:39 PMjustabill
05/08/2023, 1:24 AM