<@ULVA73B9P> Is prefect fully async ? is it bette...
# ask-marvin
a
@Marvin Is prefect fully async ? is it better to use httpx async as a client for my requests or request lib wont change much ? What do you recommend ? I have implemented few sharepoint apis that I will integrate into a flow prefect but not sure if I keep requests calls or I switch to httpx client ?
m
thought for 183.9 seconds
Short answer - Prefect 3.x supports both sync and async flows/tasks. It’s not “only async”, it’s hybrid and handles either style well. - For I/O-bound HTTP work (like SharePoint), both
requests
(sync) and
httpx
(async) work. Which is “better” depends on your concurrency needs and how you structure your flow. What I recommend - If you already have working
requests
code and your concurrency is modest, you can keep it and run tasks concurrently using Prefect’s default thread-based task runner. - If you expect higher concurrency (many simultaneous calls), want native async semantics, or plan to compose with other async code, switch to
httpx.AsyncClient
and make your tasks/flow async. Key guidelines - Don’t call blocking
requests
inside an
async
task/flow (it will block the event loop). Either: - Keep tasks sync + use the thread pool for concurrency, or - Make tasks async + use
httpx.AsyncClient
and asyncio for concurrency. - Prefect’s default runner is thread-based, which pairs fine with sync
requests
for I/O concurrency. - If you might switch to process-based or distributed execution later, construct your HTTP client inside each task (objects like clients generally aren’t picklable or serializable). Two solid patterns Option A: Keep
requests
(sync) + threads (simple, effective for moderate concurrency)
Copy code
from prefect import flow, task
from prefect.task_runners import ThreadPoolTaskRunner
import requests

@task(retries=3, retry_delay_seconds=5)
def sp_get(url: str, headers: dict):
    r = requests.get(url, headers=headers, timeout=30)
    r.raise_for_status()
    return r.json()

@flow(task_runner=ThreadPoolTaskRunner(max_workers=10))  # threads handle I/O concurrency
def ingest(urls: list[str], headers: dict):
    futures = [sp_get.submit(u, headers) for u in urls]
    return [f.result() for f in futures]
Option B: Switch to
httpx
(async) + asyncio (best scalability at higher concurrency)
Copy code
from prefect import flow, task
import httpx, asyncio

@task(retries=3, retry_delay_seconds=5)
async def sp_get(client: httpx.AsyncClient, url: str):
    r = await client.get(url, timeout=30)
    r.raise_for_status()
    return r.json()

@flow
async def ingest(urls: list[str], headers: dict):
    async with httpx.AsyncClient(headers=headers, http2=True) as client:
        results = await asyncio.gather(*(sp_get(client, u) for u in urls))
    return results
When to switch to httpx/async - You’ll issue many concurrent requests (dozens+ per flow run). - You want tighter control over timeouts, connection pooling, HTTP/2, and cancellation. - Your codebase is already leaning async or you need to interop with other async libraries. Extra tips for SharePoint APIs - Add retries with backoff for throttling. Prefect’s
@task(retries=..., retry_delay_seconds=...)
helps; you can also add jitter/backoff logic in code if needed. - Set sensible per-request timeouts. - Consider rate limiting at the application level if you see 429s. If you can share how many requests you expect per run and your deployment environment (local process, Docker, Kubernetes, distributed workers), I can recommend a precise setup (runner choice, pool size, and whether async will give you a meaningful win).