itay livni
06/04/2020, 5:06 PMFlow
has one method, update
. My concern to continue this approach is twofold: (1) Adding more code to a core piece of prefect and thus adding complexity -- That are not really essential to Flow
(2) The idea of doing direct operations on a Flow
can lead to undesirable effects (???). On the flip side is it's convenient: node_resources_fl.disjoint(fl)
flow_lst = [get_secondary_sources_fl, get_primary_sources_fl, choose_defintions_fl]
# Test disjoint
node_resources_fl = Flow("node_resources")
for fl in flow_lst:
node_resources_fl.disjoint(fl)
node_resources_fl.visualize()
The other way would be something along the lines of a module in core.utilities
. And would be structured like networkx operators.
from prefect.utilities.flow_operators import disjoint_union
new_flow = disjoint_union(get_primary_sources_fl, get_secondary_sources_fl)
Chris White
06/04/2020, 5:11 PM.disjoint
is different from .update
when the flows don’t share any tasks? Prefect can’t force separate two identical task classes so I don’t see what additional work could be achieved by a .disjoint
method
- I wonder if we should create a contrib
submodule of utilities
, and then both the complexity and maintenance burden are greatly reduced and it’s more of “This is a pattern a user found useful”itay livni
06/04/2020, 5:27 PM.disjoint
is different from .update
when the flows don’t share any tasks? Prefect can’t force separate two identical task classes so I don’t see what additional work could be achieved by a .disjoint
method"
That is correct, there is no difference. However prefect does not have a way of handling non unique parameters or, (as my understanding goes) a way to differentiate targets
with similarly named tasks
.
"`contrib` submodule of utilities
, and then both the complexity and maintenance burden are greatly reduced" I like this idea -- Its also a lightweight way to test if an idea is useful to the broader community.Chris White
06/04/2020, 5:34 PMupdate
call.
For targets / locations of similarly named tasks i recommend using either {task_run_id}
(which we will begin supporting in Core with the next release!) or {task_slug}
which is unique to each taskitay livni
06/04/2020, 5:43 PMupdate
" 🙏
As for {task_run_id}
and slug. I have reservations (for my case) when they are not human readable. But no matter. What I am really going for with these methods is to merge parameters
and other task relationship building between flows... . (1) Write many flows (2) Test (3) Join (4) Deploy as one flowChris White
06/04/2020, 5:44 PMitay livni
06/04/2020, 5:48 PMChris White
06/09/2020, 7:53 PM