Brian McFeeley
08/01/2019, 7:27 PMitems = get_list(...)
t1 = transformation.map(items)
t2 = transformation2.map(t1)
...
say we expect due to some data noise that there's a low but nonzero failure chance of one of the steps late in the series of transformations, and only for certain subsets of the data.
from a business logic perspective, the thing we'd like to happen is: every success happens as it happens, and individual task failures don't torpedo the whole pipeline, we just do some failure handling (maybe quarantining the input data in a separate bucket in s3 for analysis & notifying the data engineers, for example). The run is still a "success" in business terms if we get a large majority/plurality of tasks completed, but we do want to act on the failures by either creating new test cases or finding better filters for dirty data.
i've seen docs on state transition handling, but i'm wondering how to use that in the context of a larger flow to both 1. get a batch-wide view of the incident frequency of error transitions and 2. not tank the whole pipeline as failed, unless maybe we exceed some configurable threshold of error rate.
Does this run counter to y'alls programming model?Brett Papineau
08/01/2019, 8:35 PMBrian McFeeley
08/01/2019, 8:45 PMChris White
08/02/2019, 1:08 PMtry / except
block which, if an error is caught, raises a Prefect FAIL
signal with the inputs as the “result” so your state handler can ship them off somewhere
- going back to “default settings” -> I think playing around with triggers might be useful here; for example, we have some some_failed
type-triggers (https://docs.prefect.io/api/unreleased/triggers.html#prefect-triggers-some-failed) that allow you to only run downstream tasks if a certain proportion of upstreams fail / succeed.
- similar to the above point, you could also have a terminal task with the appropriate some_failed
trigger which will determine the flow state appropriately. This also sounds like https://github.com/PrefectHQ/prefect/issues/775 could help you as wellBrian McFeeley
08/02/2019, 3:26 PMJeremiah
08/02/2019, 5:58 PMChris White
08/05/2019, 9:58 PMMarvin
08/05/2019, 9:58 PM