Kristoffer Sahlin

10/15/2019, 10:23 AM
Hi, I have a recursive workflow problem that needs to be parallelized. Is it possible to handle it with prefect? Specifically, I have n batches of input labeled 1,...,n, and a function _merge_consecutive_batches_. Assume that n is a power of 2. The function _merge_consecutive_batches_ takes two consecutive batches, 1 and 2, 3 and 4, etc, and produces merged output for each pair, labeled say, 1_2, 3_4, … Now I want to apply _merge_consecutive_batches_ recursively until I'm down to a single output batch. For example, in the second iteration, _merge_consecutive_batches_ will take as input the two consecutive batches 1_2 and 3_4 and produce the output batch 1_2_3_4. Naturally, the jobs producing 1_2 and 3_4 have to be completed before starting this job (and they have to be run in parallel for scalability reasons). It there a solution with prefect for this scenario?