https://prefect.io logo
Title
n

Nathan Molby

03/13/2020, 2:57 PM
Question 2 of the day haha: What is the recommended way of doing asynchronous http requests? (specifically in prefect)
z

Zachary Hughes

03/13/2020, 2:58 PM
I'm a big fan of
httpx
, but if you prefer vanilla
requests
you can run that in a threadpool!
But if you're trying to run that in Prefect, it's worth noting that Prefect doesn't support async calls.
n

Nathan Molby

03/13/2020, 3:02 PM
Yeah that's the problem I've been struggling with for a bit. I'm trying to wait on a bunch of api calls, which I've previously implemented using aiohttp, but prefect doesn't support async. I'm currently trying to use the daskexecutor to potentially map and then gather the results, but I'm worried that none of the libraries will work for that.
z

Zachary Hughes

03/13/2020, 3:08 PM
Yeah, given that use case, I think your best bet is likely using synchronous requests and mapping to parallelize as desired.