Scott Moreland
11/27/2020, 1:13 PMChris White
target
keyword?Scott Moreland
11/30/2020, 2:09 PMChris White
target
is actually a special keyword that is related to file-based caching (so if the file exists, the task is not rerun); here is some documentation on `target`: https://docs.prefect.io/core/idioms/targets.html (we definitely need to expand our documentation on results and caching!)
What you can do instead is:
@task(checkpoint=True, result=MyResultType(location="{same-template-you-used-for-your-target}")
^^ that will still checkpoint your data to a templated location but will not re-use that data on subsequent runs (and will instead overwrite the data)Scott Moreland
11/30/2020, 6:21 PMChris White
checkpoint=True
+ a result location
template (note you can template these locations based on task inputs / timestamps / etc., which is sometimes useful for “overriding”): every time this task runs, it will store it’s output data to the provided location. If you ever want to “rehydrate” an upstream task’s state you’ll have to do this manually using the load_result
method on all State
objects
- `target=location_template`: when a target
is provided, the location template is first checked — if data is present at the location, it is used and the task is not rerun. If no data is present, the task runs and stores its output in the provided location. As before, you can template these locations to provide for some interesting functionality. If you ever want to force a rerun of a task, you’ll need to manually delete the data in the location yourself (this is something we do want to support automatically at some point, but it’s still under discussion)Scott Moreland
11/30/2020, 9:55 PMChris White
Marvin
11/30/2020, 10:10 PMPedro Machado
12/01/2020, 5:11 AMChris White
Pedro Machado
12/01/2020, 5:47 AM