Scott Moreland
11/24/2020, 2:42 PMJenny
11/24/2020, 3:57 PMScott Moreland
11/24/2020, 6:57 PMJenny
11/24/2020, 7:02 PMScott Moreland
11/24/2020, 7:40 PMwith Flow('my flow') as flow:
first_output = first_task()
second_output = second_task(first_output)
third_output = third_task(second_output)
if __name__ == '__main__':
flow.run()
Wondering if there is a simple way to just run second_task
, such that it reads first_output
from the checkpoint (on disk), performs the computation defined by second_task
, returns second_output
, and then exits with a success code. I guess more generally, it would be nice to run a contiguous subset of the flow graph by defining start and stop tasks if such a thing is possible with a DAG.Jenny
11/24/2020, 8:30 PMstate
to the flow.run
which could contain the state of a previous flow run that failed (and has all of the task states attached). Then the runner would only rerun the failed task and forward BUT that would also require taking the previous failed state and putting it into a Scheduled state / setting the tasks to rescheduled manually.
The reason we added that button in the UI is so it can do all that logic for you!Scott Moreland
11/24/2020, 8:39 PMJenny
11/24/2020, 8:41 PM