Jeff Yun
10/15/2019, 1:46 PMtask_runner.get_task_inputs(state, upstream_states)
but without having to run the flow)?Chris White
10/15/2019, 1:47 PMJeff Yun
10/15/2019, 1:48 PMChris White
10/15/2019, 1:50 PM__init__
(initialization)
- run through run()
, which can optionally receive inputs which always come from upstream tasks
and the Flow is responsible for tracking all dependency relationshipsJeff Yun
10/15/2019, 2:05 PMawait
their dependencies) with my own TaskRunner, instead of the current run-root-tasks-first approachprefect.context["parameters"]
3. You can't easily access non-Param task inputs passed in run
, because they come from the /final/ states of upstream tasks (and you can't just "wait" for these inputs, because you don't know which object they're from; the final upstream state object hasn't even been created)Chris White
10/15/2019, 2:30 PMtask_states
keyword argument to either flow.run
or FlowRunner.run
, which is a dictionary of task objects -> prefect state objects. This would allow you to “mock” upstream success with arbitrary resultsJeff Yun
10/15/2019, 2:31 PMA.__init__()
in self.dependencies
, and access their results in run()
or another method?Chris White
10/15/2019, 2:46 PMJeff Yun
10/15/2019, 4:04 PMmap
making 100k tasks each doing the same 1 thing, we can instead have 1 task do 100k things, i.e. submit 100k requests to an external slurm-like scheduler we use, which returns us 100k futures).Chris White
10/15/2019, 4:15 PMTaskRunner
class and use it for execution -> this might actually be the best way to go