oleander-whirlwind
05/05/2026, 10:32 AMMarvin
05/05/2026, 10:32 AMMarvin
05/05/2026, 10:33 AMThreadPoolTaskRunner(max_workers=1) only means Prefect will run one task at a time for that flow. It does not force the code inside the task to be single-threaded.
So in your example:
python
result = xr.map_blocks(
_block_stats,
da,
kwargs={"time": time, "weights": w, "stats": stats, "helper": helper},
template=template,
).compute()
whether this runs in parallel depends on xarray/dask’s execution backend, not Prefect.
What max_workers=1 means
- Prefect runs only one task concurrently
- Other Prefect tasks wait
- But that one running task may still:
- use multiple threads
- use multiple processes
- submit work to Dask
- call native libraries that parallelize internally
In practice for your case
xr.map_blocks(...).compute() usually relies on Dask for execution. So:
- if Dask is configured to use a threaded scheduler, it may use multiple threads
- if Dask is configured to use processes or a distributed cluster, it may use those
- if it’s using a synchronous scheduler, then it will run serially
So the answer is: yes, possibly parallel, but Prefect is not the thing controlling that parallelism.
Important caveat
If you use ThreadPoolTaskRunner, and your task also does heavy CPU-bound threaded work, you can run into Python GIL limitations. That means:
- Prefect thread pool concurrency and
- internal threaded compute
may contend with each other.
With max_workers=1, this is less about competition between multiple Prefect tasks, and more about whether your internal computation actually benefits from threads.
Rule of thumb
- One Prefect task at a time, but task internals can still parallelize → yes
- Will xarray.map_blocks(...).compute() parallelize? → depends on Dask scheduler/config
- If your workload is CPU-heavy, process-based or distributed execution is often better than pure threads
If you want, I can help you determine exactly whether this specific xarray call is running:
- synchronously
- with Dask threads
- with Dask processes
- on a distributed cluster
and show how to inspect/configure it.