Thread
#prefect-community
    a

    Alex Joseph

    1 year ago
    Hi, I've a mapped task creating n different subtasks. I want to persist+checkpoint the output such that if tasks completes, it doesn't need to be rerun (but the other subtasks are rerun) for the next run. I can implement this by using the
    target
    option, but I don't seem to be seeing any option to add the function arguments to template the output. For example, I'd like to have:
    @task(target="{single_pattern}_{config['a']}.txt", checkpoint=True, result=LocalResult(dir="cache/"), )
    def run_spark_job(single_pattern, config):
       ...
    I can template parameters, date etc, but I don't seem to have way to template the arguments themselves. What's the correct way to handle this situation? Thanks 🙂
    Michael Adkins

    Michael Adkins

    1 year ago
    Hi! This isn’t clearly documented there, but the inputs should be passed to the formatter as well as in https://docs.prefect.io/core/idioms/task-run-names.html#naming-task-runs-based-on-inputs
    a

    Alex Joseph

    1 year ago
    thanks @Michael Adkins - Works perfectly as (not?) advertised 🙂