@Marvin I'd like to execute some tasks in parallel but avoid using the

DaskTaskRunner

. Is there any downsides, or anything to be aware of, if I use

multiprocessing.pool.Pool.starmap()

for this?

Using

multiprocessing.pool.Pool.starmap()

for parallel task execution in Prefect has a few considerations: 1. Deadlocks and Environment Differences: Using

multiprocessing

can lead to deadlocks, especially with different spawning methods and pickling issues. This can slow down development and cause unexpected behavior. 2. Guarding Main Function: When using multiprocessing, ensure your main function is guarded with

if __name__ == "__main__":

to avoid issues with parallel processing. 3. Manual Management: Using

multiprocessing

means you have to manually handle the closing and joining of pools, which can create additional complexity. Here's a simple example using Prefect's tasks without `DaskTaskRunner`: Link: https://raw.githubusercontent.com/zzstoatzz/prefect-code-examples/main/examples/tasks/basic.py

python
from prefect import flow, task

@task(task_run_name="Adding {x} and {y}")
def add(x: int, y: int) -> int:
    return x + y

@flow
def my_flow():
    first_result = add(1, 2) 
    second_result = add(first_result, first_result)
    return add(first_result, second_result)

if __name__ == "__main__":
    my_flow()

For more information and examples, you can refer to Prefect docs: https://docs.prefect.io/latest/concepts/task-runners/ Additional context on multiprocessing issues in Prefect can be found in these GitHub issues: - Multiprocessing based TaskRunner: https://github.com/PrefectHQ/prefect/issues/12075 - Multiprocessing Deadlock: https://github.com/PrefectHQ/prefect/issues/10794 Don't hesitate to reach out if you have more questions!