let me know if this is not the best place to ask this.
But I’m curious how I might go about dynamically adjusting parameters on retries?
The use case is:
• processes taking a sum of a large number…if it overflows…I can try taking the sum with a divisor factor
• for my use case I just need a number to compare to another - which would have same divisor factor applied.
• if it fails again - I can try a larger divisor factor - up to 3 times - and then fail after the nth retry
a
Anna Geller
05/12/2022, 1:11 AM
I wonder whether this is a data flow or rather the logic of your algorithm. How would you write it in Python without Prefect?
m
Marco Pérez
05/12/2022, 1:38 AM
I would prob
1. capture the specific error
2. Augment a counter
3. if counter is <=4
4. increment the divisor factor by 10 and resubmit the query
s
Sylvain Hazard
05/12/2022, 6:39 AM
Doesn't really answer the question but couldn't just start with a relatively high divisor factor in order to avoid errors ?
upvote 1
a
Anna Geller
05/12/2022, 9:36 AM
Marco, from Prefect 1.0 perspective, you may try implementing that using looping and in Python 2.0 you can write arbitrary Python so no limits here
in Prefect 1.0 an interesting approach is that now you can use
retry_on
on your task decorator to allow tasks to retry only on a specific exception type - might be exactly what you are looking for - #5634
m
Marco Pérez
05/12/2022, 11:49 AM
Thanks everyone!
I def could try to start with a bigger number - but if the sum turns out to be a pretty small and its non-integer data types then it just doesn’t work so well.
But I’ll give that a look over. Thanks!
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.