https://prefect.io logo
d

David Ojeda

04/21/2020, 5:26 PM
I want to share the results of my long day of bug hunting. It all comes down to this screenshot: on the left, the visualization of my graph that does not work (note there are two merge nodes). On the right, the one that is fixed. I was composing three different graphs, and using
get_terminal_tasks()
to determine how to connect these graphs. Something has changed on the way
switch/merge
is managed and this created several terminal_tasks on one of my flows. The solution for me was to set the reference task of each flow and use
get_reference_task
instead of
get_terminal_tasks
Anyways, needed to share the small win after a frustrating bug-hunting session (because the bug would present itself randomly due to
set
not having an order)
🎉 5
d

Dylan

04/21/2020, 5:29 PM
Congratulations on the successful bug hunt!
z

Zachary Hughes

04/21/2020, 5:29 PM
Nice work, David!
j

Jim Crist-Harif

04/21/2020, 6:36 PM
I wonder if this is due to https://github.com/PrefectHQ/prefect/pull/2310? Calls to
switch
or
ifelse
now implicitly create and return a merged task at the end, which makes them easier to compose.
d

David Ojeda

04/21/2020, 7:21 PM
Yes I saw that PR when investigating for this issue… thanks! I think my issue may be a bit more complicated. Before, I used switch / merge to have two flows composed where only one would be executed. In the new versions of prefect, the behavior seems to be consistent, but the graph is different… it’s more difficult to understand what’s really going on… see this screenshot for example, version 0.9.4 on the right, and the latest prefect version (of today, it seems)…
In the new version, a merge is now present after the trigger tasks. It’s a bit confusing because that merge node is a final node; not used anywhere else
j

Jim Crist-Harif

04/21/2020, 7:27 PM
Yeah, I think there's a few things we should/could do here: • We could change `ifelse`/`switch` again in some way to better handle both use cases (getting the result from two options, or just branching in the graph) • We could change how we display
merge
tasks in the visualized graph • We could change the api for expressing branches to make it easier to express branches that should execute more than one task. •
An issue was filed on this new behavior already btw: https://github.com/PrefectHQ/prefect/issues/2374
d

David Ojeda

04/21/2020, 7:28 PM
Oh, great! I will follow this issue then
By the way, I noticed that the switch and ifelse conditional tasks, which now return the result of the task, are still type-annotated to return None…
đź‘Ť 1
j

Jim Crist-Harif

04/21/2020, 7:39 PM