Alex Joseph

12/03/2020, 8:06 PM
Hi, I've a mapped task creating n different subtasks. I want to persist+checkpoint the output such that if tasks completes, it doesn't need to be rerun (but the other subtasks are rerun) for the next run. I can implement this by using the
option, but I don't seem to be seeing any option to add the function arguments to template the output. For example, I'd like to have:
Copy code
@task(target="{single_pattern}_{config['a']}.txt", checkpoint=True, result=LocalResult(dir="cache/"), )
def run_spark_job(single_pattern, config):
I can template parameters, date etc, but I don't seem to have way to template the arguments themselves. What's the correct way to handle this situation? Thanks ๐Ÿ™‚


12/03/2020, 8:48 PM
Hi! This isnโ€™t clearly documented there, but the inputs should be passed to the formatter as well as in

Alex Joseph

12/04/2020, 5:14 AM
thanks @Zanie - Works perfectly as (not?) advertised ๐Ÿ™‚