Hi, Here's the sample code to the problem I mentio...
# prefect-community
Hi, Here's the sample code to the problem I mentioned yesterday were the prefect graph isn't parallelising as expected. We would expect the pre_processing function to be parallel but it isn't. The expensive computation function is being parallelised as expected. https://github.com/ralphwillgoss/prefect/tree/main/parallel
I would suggest looking at https://docs.prefect.io/core/concepts/mapping.html#prefect-approach For how to map over values
Instead of the For-Loop, you should use
on the task.
thanks @Raphaël Riel, which task btw?
I think all tasks within the loop should be remapped
ok thanks! - i think understand what you mean
Hi @Ralph Willgoss I understand the main issue, it is really strange that seemingly unrelated instances of
are not executed in parallel. I have no idea what causes it. As @Raphaël Riel suggested, using mapping is your healthiest option to parallelize the same task over a list of inputs. Sadly, mapping of mappings isn’t a first class option yet. You could do one of the mappings first, give the result to the second mapping with unmapped, and map over the second list. Return a list from the second mapped task, so you have a list of lists. Finally you can use
to generate your
task all possible combinations. Let me wrap up a basic example
thanks @emre!