Scott Zelenka08/31/2020, 1:54 PM
decorator. So it gets stuck in a perpetual loop, because the server wants the client to wait longer than what was specified in the
decorator. The Task Concurrency Limiting feature would reduce the frequency this happens, but would not catch all 429 exceptions. Sometimes this specific server gets overloaded by other traffic, and will dynamically rate-limit all traffic until it has scaled up to handle the additional traffic. I'm guessing that I'd need to write some custom retry logic within the
to handle the 429 exceptions, but curious if anyone else has a way to pipe the
from a 429 into the prefect engine's
parameter for similar rate-limited API calls?
emre08/31/2020, 2:11 PM
Next available task run had to wait 10 minutes, which is dynamically set in task.run()
class DelayTask(Task): def run(self): print(self.retry_delay) self.retry_delay = datetime.timedelta(minutes=10) raise ValueError(self.retry_delay) with Flow("delay_example") as flow: DelayTask(retry_delay=datetime.timedelta(seconds=10), max_retries=3)() flow.run()
Scott Zelenka08/31/2020, 4:18 PM
decorator, instead of the class approach .. but if the class approach works, it's easy enough to modify! Thanks emre!