Caching works with this example. With each run of the flow, prefect looks for a file "transform--2021-13-01" and if it exists, it uses the cached result. I want to add the following features:1. eWhenever, I change the source code of theWhenever I change the source code of the
return x * 2
task (say to
) I obviously don't want to use the cached result (i.e.
) as an input for the next task. Instead I want to make
recompute whenever I change the content of
. How can I do that? 2. When the result of an upstream DAG changes, I want to execute the DAG which follows from that point downwards (although the downstream tasks might also be tasks with a cache). For instance, in this example, I want
to take the new input of 20 and compute again with that input (although the result is already cached).
(will cache once for each new version of the flow) 2. The DAG will recompute downwards if it is not cached, so since you are still using task_name + today as the target on your transform task it won’t run again until the task name is different or the day has changed. Perhaps look into using
which will allow you to cache results for a particular run and if you restart it will reuse the cached results but won’t have a bearing on future runs
just seems to be the name of the flow and is not really solving the problem. Would it somehow be possible to hash the source code of a task and use that hash as a context value? Whenevery I changed something in the code, the hash would then change as well?
is also used as an argument. Any chance to get them also working with the