Hi ! I really dont understand how to rate_limit a task to prevent hitting api rate limit ?
Loup
10/30/2024, 5:15 PM
I have this task :
Copy code
@task
def request(self, endpoint: str, params: dict = {}) -> dict:
# rate_limit("tmdb-api")
self.api_key = self._get_next_api_key()
url = f"{self.base_url}/{endpoint}"
params["api_key"] = self.api_key
response = requests.get(url, params=params)
response.raise_for_status()
data = response.json()
if "success" in data and not data["success"]:
raise ValueError(f"Failed to get data from TMDB: {data}")
return data
How can I do the prevent hitting the rate limit of TMDB API ? (which is 50 requests per second
Loup
10/30/2024, 5:16 PM
I really dont understand what I have to set here :
Loup
10/30/2024, 6:14 PM
Because without the
rate_limit("tmdb-api")
my script is very fast, 14s for making 150 api requests which is way below the rate limit. But with rate_limit enable what ever the value I set its around 40s, just why ?
Loup
10/30/2024, 6:14 PM
I really dont understand
Loup
01/14/2025, 8:41 PM
I really need to fix my script who take wayy too much time because of the 'rate_limit' any idea ?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.