https://prefect.io logo
#prefect-community
Title
# prefect-community
r

Ryan Connolly

01/19/2020, 4:17 PM
we have a use case where we might have to do pretty large backfills over time when that happens... we need to be able to define dependencies that go across cycles there is an open-source project called cylc that defines this problem space well: https://cylc.github.io/ using the imperative api... this type of "cycling" can be accomplished it likely could be accomplished using the
LOOP
mechanism from the functional api... but I was having a harder time grokking that concept. I'll attach an example of the prefect-core code if anyone is interested.
e

emre

01/19/2020, 9:34 PM
I think your example is designed well. You probably either know how many backfill jobs should be run, or can learn the number pretty easily before the flow is defined, via some query or something. Either way, explicitly defining tasks and dependencies will get the most benefit out of prefect. The issue with
LOOP
is that a looping task would be represented as a single task, tasks generated in the middle of the loop cant have their own downstream tasks. You want a single model task that loops to generate
model[0], model[1]
and so forth, but attaching the model task as an upstream task to
post
or
cmp
would only attach it to the final task of the loop.
Actually, this might work. Generate a list of results using a
LOOP
ing
model
task, then use a dummy task that maps over the result list, to which
cmp
and
post
are downstream to. With a little bit of custom logic, you could handle the case of
cmp[n]
being downstream of
post[n-1]
. You could probably get rid of the dummy task as well, its too late here to properly think about it