https://prefect.io logo
r

Ralph Willgoss

10/20/2020, 3:43 PM
Hi, Here's the sample code to the problem I mentioned yesterday were the prefect graph isn't parallelising as expected. We would expect the pre_processing function to be parallel but it isn't. The expensive computation function is being parallelised as expected. https://github.com/ralphwillgoss/prefect/tree/main/parallel
r

Raphaël Riel

10/20/2020, 3:52 PM
I would suggest looking at https://docs.prefect.io/core/concepts/mapping.html#prefect-approach For how to map over values
Instead of the For-Loop, you should use
.map(letters)
on the task.
r

Ralph Willgoss

10/20/2020, 4:13 PM
thanks @Raphaël Riel, which task btw?
r

Raphaël Riel

10/20/2020, 5:04 PM
I think all tasks within the loop should be remapped
r

Ralph Willgoss

10/20/2020, 5:26 PM
ok thanks! - i think understand what you mean
e

emre

10/20/2020, 6:45 PM
Hi @Ralph Willgoss I understand the main issue, it is really strange that seemingly unrelated instances of
individual_preprocessing
are not executed in parallel. I have no idea what causes it. As @Raphaël Riel suggested, using mapping is your healthiest option to parallelize the same task over a list of inputs. Sadly, mapping of mappings isn’t a first class option yet. You could do one of the mappings first, give the result to the second mapping with unmapped, and map over the second list. Return a list from the second mapped task, so you have a list of lists. Finally you can use
flatten
to generate your
expensive
task all possible combinations. Let me wrap up a basic example
Untitled
r

Ralph Willgoss

10/21/2020, 1:25 PM
thanks @emre!
3 Views