Good evening I ran into the following problem which I find i

Question

Good evening I ran into the following problem which I find it difficult to solve My flow looks something like this And here s an example I am using localagent and this flow will be executed sequentially for each file What is the correct way to parallelize it Use multiprocessing I am more inclined to use the dask executor I looked at several examples but nowhere the flow was registered but simply executed What is the best solution to parallelize the calculations the best option I assume that each node of the executor takes a separate file for itself

bral · Answer

from prefect import task Flow case task def is running return False def get files return 1 2 3 4 task def read file file print file return True task def preprocess file file print file return True task def save file file print file return True with Flow process as flow condition = is running with case condition False as cond files = get files for file in files 0 3 r = read file file p = preprocess file file s = save file file r set downstream p p set downstream s flow register test

Kyle Moon-Wright · Answer

Hey < bral> I think the best way to parallelize your flow for your use case would be your inclination to use the `DaskExecutor` To attach it to your flow you d like to register you can attach the `DaskExecutor` to a `LocalEnvironment` for your flow like this ```from prefect import Flow from prefect environments import LocalEnvironment from prefect engine executors import DaskExecutor environment = LocalEnvironment executor=DaskExecutor flow = Flow Dask Executor Example environment=environment flow register project name= project name ```

Kyle Moon-Wright · Answer

As far as parallelizing your tasks in a flow context I d recommend checking out the dependency structure in this idiom on <https docs prefect io core idioms parallel html parallelism within a prefect flow|parallelism within a Prefect Flow>

bral · Answer

< Kyle Moon Wright> i got it thank you