Kelvin DeCosta
01/25/2023, 7:49 AM.map
for all the tasks in a batch.
While this works, it feels very hacky and isn't ideal for performance.
Any help is appreciatedZanie
Kelvin DeCosta
01/26/2023, 2:32 PMrequests
module.
At the moment the main issue is with the time taken for each task.
Sometimes the task is over within 10-20s, other times it takes 2-5 mins.Zanie
Zanie
Kelvin DeCosta
01/27/2023, 6:11 AMrequests.request
Zanie
Kelvin DeCosta
02/01/2023, 7:41 AMconcurrent.futures.ThreadPoolExecutor
, our requests
wrapper performed maybe 5-20 times faster than it does now via prefect tasks.
I'm considering migrating from requests
to httpx
.
I'd really appreciate it if you could give a rough estimate of the expected performance increase.Zanie
Zanie
Kelvin DeCosta
02/02/2023, 5:26 AM