Thread
#prefect-community
    n

    Nathan Molby

    2 years ago
    Question 2 of the day haha: What is the recommended way of doing asynchronous http requests? (specifically in prefect)
    Zachary Hughes

    Zachary Hughes

    2 years ago
    I'm a big fan of
    httpx
    , but if you prefer vanilla
    requests
    you can run that in a threadpool!
    But if you're trying to run that in Prefect, it's worth noting that Prefect doesn't support async calls.
    n

    Nathan Molby

    2 years ago
    Yeah that's the problem I've been struggling with for a bit. I'm trying to wait on a bunch of api calls, which I've previously implemented using aiohttp, but prefect doesn't support async. I'm currently trying to use the daskexecutor to potentially map and then gather the results, but I'm worried that none of the libraries will work for that.
    Zachary Hughes

    Zachary Hughes

    2 years ago
    Yeah, given that use case, I think your best bet is likely using synchronous requests and mapping to parallelize as desired.