@Marvin how can add some jitter to the start of a bunch of parallel tasks kicked off with map
m
Marvin
01/19/2025, 6:39 PM
To add jitter to the start of parallel tasks kicked off with
map
in Prefect 3.x, you can utilize the
retry_jitter_factor
option. This option introduces a random amount of time to the retry periods, helping to prevent "thundering herd" scenarios where many tasks might start at the same time.
Here's a conceptual approach:
1. Use the
retry_jitter_factor
to add variance to the base delay time.
2. Set a retry delay and apply the jitter factor to introduce random delays.
For example, if you have a retry delay of 10 seconds and set a
retry_jitter_factor
of 0.5, the delay can vary up to 15 seconds.
For more detailed implementation, you can refer to the Prefect documentation: Prefect Tasks with Jitter.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.