Jake Schmidt
12/18/2019, 4:33 PMdict
where the values are variable-length lists. I’d like to map a fanout on each key, then a fanout for each element in that key’s value, such that the operation on each value also receives the result of the “outer fanout’s” operation on its key. Would the best approach be to use a nested flow? I saw https://github.com/PrefectHQ/prefect/issues/1311 is similar...Zachary Hughes
12/18/2019, 6:00 PMJake Schmidt
12/18/2019, 6:01 PMmap
tasks, followed by a zip
task, followed by another map.for key, varlenlist in jobs.items():
key_result = key_task(key)
item_results = []
for item in varlenlist:
a = task_a(item, key_result)
b = task_b(item, key_result)
item_results.append(a)
item_results.append(b)
result = join_task(item_results)
Zachary Hughes
12/18/2019, 6:31 PMJake Schmidt
12/20/2019, 2:43 PMZachary Hughes
12/20/2019, 3:24 PMJake Schmidt
12/20/2019, 3:52 PMflow.update(…)
), so that we could at least see it in flow.visualize
? Right now visualize
seems to only compute/display the outer flow. Is there an example of the use of flow.update
?Zachary Hughes
12/20/2019, 7:19 PMChris White
12/20/2019, 7:38 PMflow.update(subflow)
will take all tasks from the subflow and add them to the parent flow. This oftentimes results in two disjoint DAGs being represented, unless you built both flows using the exact same tasks, e.g.
@task
def common_task():
...
@task
def task_one(x):
...
@task
def task_two(y):
...
with Flow("one") as flow_one:
a = task_one(common_task)
with Flow("two") as flow_two:
b = task_two(common_task)
flow_two.update(flow_one)