https://prefect.io logo
#prefect-community
Title
# prefect-community
d

Dragan

06/01/2022, 8:09 AM
Hi all, does anyone know if there is a parameter for the job that can be set to “allow to fail” Not to get confused, the use case is different We are running some validations during our complete etl flow, orchestrated in Prefect one of the flows is this validations, but if any of the validation is false, the flow will fail and also the entire job and that we would like to avoid so our steps are, we trigger one flow consisting on 5 different flows in cycle: 1. calculations finish 2. load finish 3. data transfer finish 4. validations finish (found one error but it finished the job) - after that everything fail, next job is not triggered we would like to be able to trigger the next job even if the validation founds some errors, since it will trigger Slack webhook and inform us, and we can sort the error, but our job will continue is something like this possible
a

Amanda Wee

06/01/2022, 8:30 AM
What if you set the reference tasks such that if these validations fail, the reference tasks can still be successful? Then the flow will succeed even if these validations fail.
🙌 1
s

Sylvain Hazard

06/01/2022, 8:50 AM
As I understand it, you're running a flow of flows configuration. If that's the case, you can use
wait_for_flow_run(flow, raise_final_state=False)
to have the "master flow" task succeed regardless of the state of the subflow. More info here.
🙌 1
d

Dragan

06/01/2022, 9:39 AM
perfect, we will try it out thanks
5 Views