I wonder if there is a recommended way to clear ta...
# prefect-community
e
I wonder if there is a recommended way to clear task input cache? Details below. Suppose I've set up and ran a task with
cache_key_fn=task_input_hash
and
cache_expiration=timedelta(days=1)
. Now for some reason I want to clear cache and run the task again (suppose I've changed the logic in a module I'm using from the task). My options (including non-working): 1. Deleting cache files doesn't work. If I run a task after deleting files I get an error. Prefect doesn't check if the file exists before deciding on using cache. 2. Deleting flow runs or task runs does work. But it's not convenient for two reasons: a. I should delete history of runs. b. I have to remember which flow/task run to delete or delete all of them. 3. Adding extra parameter
cache_num
to task should work. I can just change it's value every time I want to avoid using cache. But it generates extra boilerplate code and I have to change flow/task code to change parameter value. It looks like a dirty duct tape to me) 4. Changing data in OrionDB maybe possible, but I'm sure that it's strongly not recommended. And I'd like to avoid it. Maybe there is a better option I don't know about? A perfect solution would be CLI or API with flow/task name as an input parameter.
1
1
t
Hey together, we are having exactly the same challenge at the moment. Just dig a little bit into Gitlab repo and found this function: https://github.com/PrefectHQ/prefect/blob/240c8c45a9a4cd46112bc3b90ac931c7e605cd19/src/prefect/orion/orchestration/core_policy.py#L212
Hey, we are having the same challenge right now. We are interested in a proper solution without manipulating the Orion DB directly or introducing another input parameter to have something that changes from run to run. I dug a bit into Gitlab repo and found this function linked below. As I understand, Orion decides whether to execute or use cache. As a user, I have two possible interfaces in mind: 1. Having a boolean decorator like "cache_force_execution" that can be set true at runtime via "with_options". -> This approach overrides the decision made by Orion during execution. 2. Orion API call that sets expiration = now-1 for a given flow/task. -> This approach gives you the control over cached results. https://github.com/PrefectHQ/prefect/blob/240c8c45a9a4cd46112bc3b90ac931c7e605cd19/src/prefect/orion/orchestration/core_policy.py#L212
👍 1
I would prefer Option 1: With option 2, you invalidate all cached results for a flow/task. You are not able to invalidate the cached result only for a given set of input parameters directly.
j
Hi Evgeny and Tobias. Thank you for raising. There’s an open issue for this feature request. I encourage you to chime in there to let the team know about your desires and so you can follow the progress. https://github.com/PrefectHQ/prefect/issues/7315
❤️ 2