hi, I've found the conditional tasks to be rather confusing. In particular I was hoping that if you ...

Daniel Manson

10/20/2021, 1:31 PM

hi, I've found the conditional tasks to be rather confusing. In particular I was hoping that if you have something like this...

Copy code

a = task_a()
test_value = task_get_test_value()
with case(test_value, 'test-against-this'):
  b = task_b()
  b.set_upstream(a)

...then task

would fail if

fails regardless of whether the

case

block is in the skipped or non skipped state. However that doesn't seem to be the way it works - rather if the

case

block skips, then it 'hides' whether or not

succeeded.

Daniel Manson

10/20/2021, 1:33 PM

i also found it confusing that if you wanted to have a

task_c

and set

to be upstream, then you need to be very careful around not propagating the skipping. That seems at odds with the idea of a

with

block isolating logic..it's actually leaked out

Daniel Manson

10/20/2021, 1:34 PM

i'm now trying to write my own verison of the

case

class to see if i can accomplish what i'm after, but i'm finding i'm running into gaps in documentation, e.g. around the

trigger

function on a task

Kevin Kho

10/20/2021, 1:47 PM

Hey @Daniel Manson, I think this is because the case is a task and it propagates

SKIP

if the case is not met.

task_b

then has two upstream tasks with

task_a

and

case

task_a

will be

FAILED

and

case

will be

SKIPPED

and

SKIPPED

propagates. You can try with the

skip_on_upstream_skip

flag

Copy code

@task(skip_on_upstream_skip=True)
def task_b():
    ...

a = task_a()
test_value = task_get_test_value()
with case(test_value, 'test-against-this'):
  b = task_b()
  b.set_upstream(a)

Daniel Manson

10/20/2021, 1:49 PM

isn't

True

the default for

skip_on_upstream_skip

Kevin Kho

10/20/2021, 1:53 PM

Ah sorry. You’re right. Should be False

Daniel Manson

10/20/2021, 1:54 PM

but wouldn't that mean

is never skipped?

Daniel Manson

10/20/2021, 1:54 PM

what we want is it to propagate failures of

through

, but allow

to be skipped if the

case

is false

Daniel Manson

10/20/2021, 1:55 PM

i'm not sure why that doesn't happen already, because the defualt

trigger

all_successful

and `b`'s upstream tasks are [

case

], so even if

case

is skipped,

should fail if

has failed, no?

Kevin Kho

10/20/2021, 1:56 PM

Let me try this out a bit

Kevin Kho

10/20/2021, 2:04 PM

is never

SKIPPED

but it won’t run with

TriggerFailed

. Still not ideal. Still looking into it.

Daniel Manson

10/20/2021, 2:04 PM

thanks

Kevin Kho

10/20/2021, 2:13 PM

Ok I am starting to believe this is indeed a gap in the triggers because the main issue is the two upstream tasks with

FAILED

and

SKIPPED

will

SKIP

the next task and we need to get failed. I think I will open an issue to see how we can address that. Would you like me to open an issue or do you want to?

Daniel Manson

10/20/2021, 2:14 PM

you are welcome to - that would be great

Kevin Kho

10/20/2021, 2:20 PM

On it

Kevin Kho

10/20/2021, 2:28 PM

I dunno how feasible this is honestly: https://github.com/PrefectHQ/prefect/issues/5071

Daniel Manson

10/20/2021, 2:32 PM

thanks

Daniel Manson

10/20/2021, 2:32 PM

do you have any vauge idea how long something like this might take to go into a release (if at all)? are we talking days or weeks?

Kevin Kho

10/20/2021, 2:34 PM

I don’t because I can’t even picture the solution for this one. Or maybe the core team just details a better workaround than me.

👍 1

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.