we have a use case where we might have to do pretty large ba

Question

we have a use case where we might have to do pretty large backfills over time when that happens we need to be able to define dependencies that go across cycles there is an open source project called cylc that defines this problem space well <https cylc github io > using the imperative api this type of cycling can be accomplished it likely could be accomplished using the `LOOP` mechanism from the functional api but I was having a harder time grokking that concept I ll attach an example of the prefect core code if anyone is interested

emre · Answer

I think your example is designed well You probably either know how many backfill jobs should be run or can learn the number pretty easily before the flow is defined via some query or something Either way explicitly defining tasks and dependencies will get the most benefit out of prefect The issue with `LOOP` is that a looping task would be represented as a single task tasks generated in the middle of the loop cant have their own downstream tasks You want a single model task that loops to generate `model 0 model 1 ` and so forth but attaching the model task as an upstream task to `post` or `cmp` would only attach it to the final task of the loop

emre · Answer

Actually this might work Generate a list of results using a `LOOP` ing `model` task then use a dummy task that maps over the result list to which `cmp` and `post` are downstream to With a little bit of custom logic you could handle the case of `cmp n ` being downstream of `post n 1 ` You could probably get rid of the dummy task as well its too late here to properly think about it