Hi, Here's the sample code to the problem I mentioned yesterday were the prefect graph isn't parallelising as expected. We would expect the pre_processing function to be parallel but it isn't. The expensive computation function is being parallelised as expected. https://github.com/ralphwillgoss/prefect/tree/main/parallel

I would suggest looking at https://docs.prefect.io/core/concepts/mapping.html#prefect-approach For how to map over values

thanks @Raphaël Riel, which task btw?

I think all tasks within the loop should be remapped

ok thanks! - i think understand what you mean

Hi @Ralph Willgoss I understand the main issue, it is really strange that seemingly unrelated instances of

individual_preprocessing

are not executed in parallel. I have no idea what causes it. As @Raphaël Riel suggested, using mapping is your healthiest option to parallelize the same task over a list of inputs. Sadly, mapping of mappings isn’t a first class option yet. You could do one of the mappings first, give the result to the second mapping with unmapped, and map over the second list. Return a list from the second mapped task, so you have a list of lists. Finally you can use

flatten

to generate your

expensive

task all possible combinations. Let me wrap up a basic example

thanks @emre!