Gregory Hunt
01/24/2025, 7:10 PMConcurrentTaskRunner
use fork or spawn for multiprocessingMarvin
01/24/2025, 7:10 PMConcurrentTaskRunner
in Prefect 2.x uses a thread pool maintained by anyio
for concurrency. It does not use multiprocessing with fork or spawn; instead, it allows tasks to switch when blocking on IO. If you are looking for parallel execution using multiprocessing, you might consider using the DaskTaskRunner
or RayTaskRunner
for distributed task execution. For more details, you can visit the Prefect 2.x task runners documentation.