Hi Beginner question here for prefect 1 0 How would you advi

Question

Hi Beginner question here for prefect 1 0 How would you advise the following type of pattern be implemented I will have a number of tasks some of which should have error handling tasks and the final task should only be invoked if the error handlers have either a not been triggered or b succeeded in correcting the error I ve attached a schematic where green arrows represent successes and red failures If there is no error handler attached to a task then a failure should fail the flow as well

Anna Geller · Answer

you can use triggers to do that <https docs prefect io core about prefect next steps html>

Anna Geller · Answer

thanks for the image it helps a lot understanding what you re trying to build

Anna Geller · Answer

specifically you d need `any successful` `any failed`

Oscar Krantz · Answer

thanks for the quick response I tried with any failed for my t fail task but when the task succeeds it reports a TriggerFailed which causes the flow to report failure as well Maybe I m misunderstanding the use

Kevin Kho · Answer

The flow state is dependent on the <https docs prefect io core concepts flows html reference tasks|reference task> so maybe you can define it to the the final one

Oscar Krantz · Answer

Ah I see With this flow what would be an appropriate trigger for the final task I would need it to trigger when t3 was successful and for t1 t2 when either the error handler or the task was successful t fail t success I ve set all the boxes which connect to final task as upstream tasks right now

Kevin Kho · Answer

It s very hard to do stuff like if a succeeds but b fails with the existing triggers I am not 100 sure but you might have an easier time using a <https docs prefect io core concepts notifications html state handlers|state handler> for the failure logic

Oscar Krantz · Answer

Can I use state handlers to launch a new task in response to failures Something like ```def state handler if task failed new = other task return new state else continue on```

Kevin Kho · Answer

That you can t do Like you can do `task run ` but it s just normal Python It s not a task with retries

Oscar Krantz · Answer

Hm okay That might present a trouble for us using prefect then Typically we d need this type of flow for external file deliveries The first operations here would be polling for files and if not found then we implement some error handler which can grab an old file or such Some files if not found should terminate the flow t3 here The final operation a database load for instance should only commence if all files were either found or the error handlers successfully grabbed an old file and this needs to be transparent in the UI

Kevin Kho · Answer

Let me think for a bit

Kevin Kho · Answer

I think you need an additional task like this ```from prefect import task Flow import prefect from prefect triggers import any failed any successful task def task one x if x == False raise ValueError return x task trigger=any successful def task one success TASK ONE SUCCESS return 1 task trigger=any failed def task one failed TASK ONE FAILED return 2 task def task two x if x == False raise ValueError return x task trigger=any successful def task two success TASK TWO SUCCESS return 1 task trigger=any failed def task two failed TASK TWO FAILED return 2 task trigger=any successful def merge return True task def final task FINAL TASK return 1 with Flow as flow a = task one False a s = task one success upstream tasks= a a f = task one failed upstream tasks= a b = task two True b s = task two success upstream tasks= b b f = task two failed upstream tasks= b a res = merge upstream tasks= a s a f b res = merge upstream tasks= b s b f final task upstream tasks= a res b res flow run ```

Oscar Krantz · Answer

That seems to work very well Nice thinking with the merge task I ll try a solution using `case` as well Thanks