Ron Gross
12/22/2020, 8:03 AMwith case
and upstream_tasks
together
we have a set of 3 tasks that need to run one after the other (no values shared between them)
A -> B -> C
we also would like to have the ability to run only:
A -> C
that means that B
should go inside with case
, but then what happens is that C
is skipped, because it has upstream_tasks
related to B
any solutions available?Jim Crist-Harif
12/22/2020, 3:47 PMskip_on_upstream_skip=False
on C
. Note that this means that C
will never be skipped, even if A
is skipped. This may not matter for your use case.
• Encode the skipping of B
inside the B
task itself. B
always runs, but some parameter to it skips most of the internals, returning quickly. Simple, and doesn't require any complicated task munging.
• Wrap B
with a merge
task (with a no-op), so that some "B" always runs. This means that C
won't be skipped unless A
is skipped, but is a bit more work. Something like:
from prefect.tasks.control_flow import merge
from prefect import case, task
@task
def noop():
pass
with case(cond, True):
b1 = B()
with case(cond, False):
b2 = noop()
b = merge(b1, b2)
Jim Crist-Harif
12/22/2020, 3:49 PMprefect-community
channel - prefect-server
is intended for conversations about using Prefect Server)Ron Gross
12/22/2020, 5:42 PM