Hello everyone, I have a question about how to handle failures in flow runs: I have a basic ETL pipeline set up as a flow that is scheduled to execute daily, and it ingests data from about a 100 different endpoints. It is expected that the ingest for most of those endpoints will be successful, but also that there will always be several that are unsuccessful. But since some are unsuccessful, Prefect shows the entire flow run as a failure. Is there a way to change that? I'd prefer that it only show as a failure if none of the ingests are successful, because it's set up to alert people who use this about failures. The Flow object in Python is set up sort of like this

source_ids = ["source_1", "source_2", "source_3"]
with Flow("my_flow") as flow:
    for source_id in source_ids:
        data = extract(source_id)
        data = transform(data)
        load(data)
    return flow

Thanks in advance!

Hi @Alexander Kloumann. Thanks for asking! If you’d be so kind as to post in the #CL09KU1K7 channel instead and put the flow code in a thread to keep the channel clean, that would probably get more eyes on your question. It looks like this is Prefect 1 code and I don’t have an answer from you, but I bet someone else in the community might. 🙂

Thanks @Jeff Hale! I'll put it in that channel now.