https://prefect.io logo
k

Kyle

07/10/2023, 5:02 AM
^ i think the solution that would work best is using redis with “ratelimiter” in python. i dont see anything in prefect docs so i assume i need something custom.
Copy code
import redis 
from ratelimiter import RateLimiter

# Connect to your Redis server
redis_conn = redis.Redis(host='localhost', port=6379, db=0)

rate_limiter1 = RateLimiter(redis_conn, max_calls=500, period=10, name='ratelimit1')
rate_limiter2 = RateLimiter(redis_conn, max_calls=1000, period=60, name='ratelimit2')

with rate_limiter1:
    try:
        response = requests.get(BASEURL)
    except Exception as e:
        msg = f"[{ticker}] API request failed: {e}"
        raise Exception(msg)