I think in general, when you use mapping, you need to template the location of the result so that each mapped task gets a separate file. Or you are you talking about
cache_for
and
cache_validator
?
Kevin Kho
12/14/2021, 3:35 PM
Also the
apply_map
is just a function that adds tasks to your Flow so it should be those tasks that have checkpointing
t
Tom Shaffner
12/14/2021, 4:28 PM
I was talking about the
cache_for
, sounds like templating is the way to go.
k
Kevin Kho
12/14/2021, 4:37 PM
I think templating should help you split out the files yep
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.