@Marvin is there a way to have allow_failure to wait until the task has completed all the retries?

It appears that you've stumbled upon a known issue related to

allow_failure

and task retries within Prefect. Here is the summary of the problem: - When a task fails due to exceeding

timeout_seconds

, even after retrying (

retries

), the following task when submitted with

wait_for=allow_failure(result)

does not execute as expected. This behavior is specific to failures caused by timeout/retries scenarios. This issue has been reported on GitHub in issue #8029. It affects Prefect version 2.7.4, and from the user contributed bug report, it seems that this might not necessarily be an intentioned behavior and could be a bug. I would suggest tracking the progress of this issue on the Prefect GitHub page to receive updates about fixes or workarounds. If this is causing a significant problem in your workflow and it is critical, consider reaching out on the issue for potential expedited support. Remember that open-source projects operate on a combination of maintainers' timelines and community engagement, so response times can vary.