Question 2 of the day haha: What is the recommended way of doing asynchronous http requests? (specifically in prefect)
03/13/2020, 2:58 PM
I'm a big fan of
, but if you prefer vanilla
you can run that in a threadpool!
But if you're trying to run that in Prefect, it's worth noting that Prefect doesn't support async calls.
03/13/2020, 3:02 PM
Yeah that's the problem I've been struggling with for a bit. I'm trying to wait on a bunch of api calls, which I've previously implemented using aiohttp, but prefect doesn't support async. I'm currently trying to use the daskexecutor to potentially map and then gather the results, but I'm worried that none of the libraries will work for that.
03/13/2020, 3:08 PM
Yeah, given that use case, I think your best bet is likely using synchronous requests and mapping to parallelize as desired.