alex
07/07/2020, 4:27 PMwith Flow("My flow", schedule=schedule) as flow:
for source in all_sources:
source.bind(
## parameters for my run() function here
)
res = [source() for feed in all_sources]
In the schematic and gantt chart I have a dag with all the sources
leading to a list, which is expected but I also duplicated source
tasks without any edgesZachary Hughes
07/07/2020, 4:33 PMalex
07/07/2020, 4:40 PMbind()
call resolved that issue, I think maybe I was just using it incorrectly?Zachary Hughes
07/07/2020, 4:59 PMbind
creates a keyed edge-- I'm wondering if there's some peculiarity in visualization between keyed edges and non-keyed edges.alex
07/07/2020, 5:29 PMclass DerivedMeta(SignatureValidator, type(ABC)):
pass
class BaseSource(Task, metaclass=DerivedMeta):
# has run method w custom parameters
class DerivedClasses(BaseSource):
# no run implementation, uses parent's
all_sources
is a list[DerivedClasses]Zachary Hughes
07/07/2020, 6:16 PMres = [source() for feed in all_sources]
, you create a separate set of tasks without edges. That would explain the duplicate set of tasks you were seeing.alex
07/07/2020, 7:11 PMbind
, what would the best way to achieve what I am trying to do be? Would just using res = [source(**run_kwargs) for feed in all_sources]
instead of bind() be ok?Jim Crist-Harif
07/07/2020, 7:34 PMbind
can be useful, but we recommend using the functional api when possible.