<@ULVA73B9P> when i use task_runner=ThreadPoolTask...
# ask-marvin
o
@Marvin when i use task_runner=ThreadPoolTaskRunner(max_workers=1) in my flow, is my task then able to use paralelled processing? template = xr.DataArray( template_data, dims=("stats", "y", "x"), coords={"stats": stats, "y": da["y"], "x": da["x"]}, attrs=da.attrs, ) - result = xr.map_blocks( _block_stats, da, kwargs={"time": time, "weights": w, "stats": stats, "helper": helper}, template=template, ).compute()
m
thought for 98.4 seconds
Yes — inside that one Prefect task, your code can still do its own parallel work.
ThreadPoolTaskRunner(max_workers=1)
only means Prefect will run one task at a time for that flow. It does not force the code inside the task to be single-threaded. So in your example:
Copy code
python
result = xr.map_blocks(
    _block_stats,
    da,
    kwargs={"time": time, "weights": w, "stats": stats, "helper": helper},
    template=template,
).compute()
whether this runs in parallel depends on xarray/dask’s execution backend, not Prefect. What
max_workers=1
means
- Prefect runs only one task concurrently - Other Prefect tasks wait - But that one running task may still: - use multiple threads - use multiple processes - submit work to Dask - call native libraries that parallelize internally In practice for your case
xr.map_blocks(...).compute()
usually relies on Dask for execution. So: - if Dask is configured to use a threaded scheduler, it may use multiple threads - if Dask is configured to use processes or a distributed cluster, it may use those - if it’s using a synchronous scheduler, then it will run serially So the answer is: yes, possibly parallel, but Prefect is not the thing controlling that parallelism. Important caveat If you use
ThreadPoolTaskRunner
, and your task also does heavy CPU-bound threaded work, you can run into Python GIL limitations. That means: - Prefect thread pool concurrency and - internal threaded compute may contend with each other. With
max_workers=1
, this is less about competition between multiple Prefect tasks, and more about whether your internal computation actually benefits from threads. Rule of thumb - One Prefect task at a time, but task internals can still parallelize → yes - Will
xarray.map_blocks(...).compute()
parallelize?
→ depends on Dask scheduler/config - If your workload is CPU-heavy, process-based or distributed execution is often better than pure threads If you want, I can help you determine exactly whether this specific
xarray
call is running: - synchronously - with Dask threads - with Dask processes - on a distributed cluster and show how to inspect/configure it.