Jullian Bellino04/04/2022, 8:38 AM
However it looks like this could block the other running tasks because the runner is not released. Now let's imagine we only rely on a single local agent, with maximum 3 tasks in parallel, in case some S3 partitions are late for any reason, it could quickly block all of the others tasks. Do you have any better way to achieve this ?
raise RETRY("Partition not available yet, retrying in 1 hour.", start_time=pendulum.now().add(hours=1))
Anna Geller04/04/2022, 9:26 AM
Jullian Bellino04/04/2022, 9:49 AM
Anna Geller04/04/2022, 10:43 AM