Robert Banick05/04/2023, 11:13 PM
parameter just released (!). Is is possible to pass in additional arguments to function so invoked, using objects generated during the course of a flow? Like so:
My use case is wiping very large amounts of data downloaded for temporary processing as a part of my ETLs. Leaving this lying around on temporary storage is expensive and best avoided. The path name for the directory to wipe depends on the ETL so invoked and so a static function won’t cut it; I need to vary the path I apply
def cleanup(path_name: str): shutil.rmtree(path_name) def crash_hook(path_name, flow, flow_run, state): cleanup(path_name) @flow(on_crashed=[crash_hook]) def my_flow(): # stuff path_name = other_function()
to. Thanks for pushing this awesome feature, it directly responds to a problem I was just about to address. Very cool!
Abhinav Chordia05/05/2023, 2:29 AM
justabill05/05/2023, 12:10 PM
as a string block so it can be retrieved.
Robert Banick05/05/2023, 3:39 PM
justabill05/08/2023, 1:24 AM