https://prefect.io logo
a

Alex Joseph

12/03/2020, 8:06 PM
Hi, I've a mapped task creating n different subtasks. I want to persist+checkpoint the output such that if tasks completes, it doesn't need to be rerun (but the other subtasks are rerun) for the next run. I can implement this by using the
target
option, but I don't seem to be seeing any option to add the function arguments to template the output. For example, I'd like to have:
Copy code
@task(target="{single_pattern}_{config['a']}.txt", checkpoint=True, result=LocalResult(dir="cache/"), )
def run_spark_job(single_pattern, config):
   ...
I can template parameters, date etc, but I don't seem to have way to template the arguments themselves. What's the correct way to handle this situation? Thanks 🙂
z

Zanie

12/03/2020, 8:48 PM
Hi! This isn’t clearly documented there, but the inputs should be passed to the formatter as well as in https://docs.prefect.io/core/idioms/task-run-names.html#naming-task-runs-based-on-inputs
a

Alex Joseph

12/04/2020, 5:14 AM
thanks @Zanie - Works perfectly as (not?) advertised 🙂