Sorry if this has already been answered - I just couldn't find it in the docs - perhaps I've just ha...

Jacob Bedard

11/03/2021, 4:52 AM

Sorry if this has already been answered - I just couldn't find it in the docs - perhaps I've just had a really long day and can't think of the right search terms. What's the best practice for forcing a failure if a task or flow doesn't pass a custom validation I've made? For example, if the rows loaded into a destination don't match the row count that was supposed to be loaded? Stuff like that happens with Salesforce all the time and their callbacks aren't very flexible, so I like to check to see if rows loaded are what I expected.

Jacob Bedard

11/03/2021, 4:56 AM

Should I call raise and use the signals functionality?

Kevin Kho

11/03/2021, 5:02 AM

Hi @Jacob Bedard, if this were me I’d check the condition and raise a state like you suggest.

Copy code

@task
def task_with_check(...):
    if n_rows != 1000:
        raise FAILED("msg")
    return data

Kevin Kho

11/03/2021, 5:02 AM

If you need the signals docs. They are here

Kevin Kho

11/03/2021, 5:02 AM

Note that raising FAILED respects retries. If you don’t want retries, use ENDRUN

🙏 1

Jacob Bedard

11/03/2021, 5:12 AM

Ok, so I wasn't completely off course. Thanks Kevin!

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.