Hello 🙋 let's say i have a mapped task

X

that spawn several "children" tasks

Y1

...

Yn

let's say some of them are stuck on

Running

as a finished state - but i wanna guarantee the next phase in the flow executes correctly and ignores the few "hung" tasks -- what's the (default) behavior for if i set those tasks (e.g.

Y3

,

Y17

and

Y91

) to

Skipped

? would the next task that depends on them still get executed (even if it has the default

all_successful

trigger?) the reason i'm asking about

Skipped

is because i wanna avoid a

None

response flowing downstream from these tasks

there is no solution for it when the pod dies; we discussed it already in previous issues and on GitHub. I understand your pain here but we don't have the capacity to tackle it in 1.0, we are aware and we'll tackle it in 2.0

in pseudo code it would be:

results = X.map(inputs)
final_output = Z(result)

there are 2 things: 1. To determine the flow run state, you can set reference tasks https://docs.prefect.io/core/concepts/flows.html#reference-tasks 2. To manually change task run states, you can do that either through the UI or use the mutation set_task_run_states

i know how to change the state. the question was how would downstream tasks behave if some of the upstream tasks are

Skipped

- would they still run ?

it depends on triggers, Skipped is considered the same as Success

@Anna Geller finally, all 580 tasks finished except 3 that failed and cannot be salvaged i set those to

Skipped

and then restarted the flow --- and then the downstream task was skipped as well and i got :

Upstream task was skipped; if this was not the intended behavior, consider changing `skip_on_upstream_skip=False` for this task.

on the downstream task is there any other state i can put them on that would force the downstream task to run for this existing flow run ? (running the entire flow takes 24h so i don’t wanna redo it) or is there another way to force the downstream task to run?

not sure what's the best way, perhaps manually starting those tasks via a separate flow triggered with flow.run()?

@Anna Geller but it’s a downstream task, if i start a new flow or run them manually they won’t have the inputs from upstream..? basically the flow is: 1. query snowflake 2. save to CSV 3. split CSV to (580) chunks of size 100 4. map them to a processing task 5. combine all 580 outputs to a single CSV 6. upload

Congratulations, glad you got it working

@Anna Geller thanks - and sorry for the long (😳) writeup but - but- some feedback for prefect, based on this experience of running a heavy flow (in case not all of these are addressed in 2.0 already): • my life would have made much simpler if the GraphQL tool allowed to modify (existing) task's triggers or other attributes (e.g. , if i could change the task's attribute to

skip_on_upstream_skip=False

. If the goal is to prevent negative engineering - why not give users more power over their existing flow run - since changing the code would necessarily obligate a new flow run which could be costly (both in time and in money or other resources) • ultimately, these 3 tasks failed due to various "weaknesses" in my own code and/or in Prefect - namely: ◦ my shell script did not create an output when the input was empty (this then led the task to fail when it tried to read the output file and provide it as a result of the task) ◦ their input was empty because i had a small bug in my code, in the upstream task which splits the CSVs ◦ even if i wanted to force the failed tasks to be marked (artificially) as

Successful

this would not have been enough because the downstream task assumed all its inputs are valid (e.g. no

None

) ◦ the downstream task did not have

skip_on_upstream_skip=False

set on it, so i could not skip the upstream tasks ◦ the trigger for the downstream task was

all successful

(which is the default) so it did not tolerate even 3 failures out of 580 • I started to write here various ideas for how Prefect could be more "fool-proof" but honestly i'm not sure i have a clear and universally good idea at the moment - most of the ideas i had basically involved users themselves taking part in some defensive engineering (which you like to call negative engineering ) -- which is exactly the kind of thing Prefect aims to eliminate. • but, there must be some ways for Prefect to be more tolerant to issues like the ones i described, and make it easier to recover from catastrophes without repeating work and wasting resources... 🤔 if users have to write 100% correct Prefect flows always ahead of time (and a single wrong attribute or trigger could make the entire thing fail after tons of time & resources have been spent running the flow) --- then this just pushes the negative engineering problem one level "upwards", from the business logic to the orchestrating-code level --- without eliminating it: instead of making sure my own code can deal with failures, i'm making sure prefect can deal with failures, since a single misconfiguration could spell disaster (in this case, i only managed to salvage it with some "hacking" - i.e. artificially manipulating checkpointed data on

S3

)

Thanks for the feedback, I'm sure we can tackle many of the pain points in Prefect 2.0