Hi all, does anyone know if there is a parameter for the job that can be set to “allow to fail”
Not to get confused, the use case is different
We are running some validations during our complete etl flow, orchestrated in Prefect
one of the flows is this validations, but if any of the validation is false, the flow will fail and also the entire job and that we would like to avoid
so our steps are, we trigger one flow consisting on 5 different flows in cycle:
1. calculations finish
2. load finish
3. data transfer finish
4. validations finish (found one error but it finished the job) - after that everything fail, next job is not triggered
we would like to be able to trigger the next job even if the validation founds some errors, since it will trigger Slack webhook and inform us, and we can sort the error, but our job will continue
is something like this possible
a
Amanda Wee
06/01/2022, 8:30 AM
What if you set the reference tasks such that if these validations fail, the reference tasks can still be successful? Then the flow will succeed even if these validations fail.
🙌 1
s
Sylvain Hazard
06/01/2022, 8:50 AM
As I understand it, you're running a flow of flows configuration. If that's the case, you can use
wait_for_flow_run(flow, raise_final_state=False)
to have the "master flow" task succeed regardless of the state of the subflow. More info here.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.