Hi everyone ! Relatively new to Prefect, we're currently using this (great!) tool to orchestrate our ELT process (with Airbyte & DBT). Until yesterday and since a few weeks, everything was working great but this night, an odd error occurred on my `AirbyteConnectionTask`:
•
JOB XXX failed
•
Final state : Success
All my tasks were set as reference, so I really don't know how to explain why this flow showed no trace of error (taskbar was green, etc.). Do you have any clues to explain this behaviour?
a
Anna Geller
01/24/2022, 11:25 AM
That’s indeed quite weird. I checked the Airbyte sync task and it looks really good when it comes to polling for status - see this line. But I believe there is a bug when it comes to handling a task run failure - I think it would need to raise an exception when the job status is failed in order to mark this task run as failed rather than only logging the error.
I will open a Github issue to discuss with other engineers. Thanks for raising this!
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.