Akram Gharbi
02/18/2026, 2:45 PMMarvin
02/18/2026, 2:45 PMMarvin
02/18/2026, 2:48 PMrequests (sync) and httpx (async) work. Which is “better” depends on your concurrency needs and how you structure your flow.
What I recommend
- If you already have working requests code and your concurrency is modest, you can keep it and run tasks concurrently using Prefect’s default thread-based task runner.
- If you expect higher concurrency (many simultaneous calls), want native async semantics, or plan to compose with other async code, switch to httpx.AsyncClient and make your tasks/flow async.
Key guidelines
- Don’t call blocking requests inside an async task/flow (it will block the event loop). Either:
- Keep tasks sync + use the thread pool for concurrency, or
- Make tasks async + use httpx.AsyncClient and asyncio for concurrency.
- Prefect’s default runner is thread-based, which pairs fine with sync requests for I/O concurrency.
- If you might switch to process-based or distributed execution later, construct your HTTP client inside each task (objects like clients generally aren’t picklable or serializable).
Two solid patterns
Option A: Keep requests (sync) + threads (simple, effective for moderate concurrency)
from prefect import flow, task
from prefect.task_runners import ThreadPoolTaskRunner
import requests
@task(retries=3, retry_delay_seconds=5)
def sp_get(url: str, headers: dict):
r = requests.get(url, headers=headers, timeout=30)
r.raise_for_status()
return r.json()
@flow(task_runner=ThreadPoolTaskRunner(max_workers=10)) # threads handle I/O concurrency
def ingest(urls: list[str], headers: dict):
futures = [sp_get.submit(u, headers) for u in urls]
return [f.result() for f in futures]
Option B: Switch to httpx (async) + asyncio (best scalability at higher concurrency)
from prefect import flow, task
import httpx, asyncio
@task(retries=3, retry_delay_seconds=5)
async def sp_get(client: httpx.AsyncClient, url: str):
r = await client.get(url, timeout=30)
r.raise_for_status()
return r.json()
@flow
async def ingest(urls: list[str], headers: dict):
async with httpx.AsyncClient(headers=headers, http2=True) as client:
results = await asyncio.gather(*(sp_get(client, u) for u in urls))
return results
When to switch to httpx/async
- You’ll issue many concurrent requests (dozens+ per flow run).
- You want tighter control over timeouts, connection pooling, HTTP/2, and cancellation.
- Your codebase is already leaning async or you need to interop with other async libraries.
Extra tips for SharePoint APIs
- Add retries with backoff for throttling. Prefect’s @task(retries=..., retry_delay_seconds=...) helps; you can also add jitter/backoff logic in code if needed.
- Set sensible per-request timeouts.
- Consider rate limiting at the application level if you see 429s.
If you can share how many requests you expect per run and your deployment environment (local process, Docker, Kubernetes, distributed workers), I can recommend a precise setup (runner choice, pool size, and whether async will give you a meaningful win).