@Marvin in prefect 3, how to define a cache policy based on task inputs and source?

In Prefect 3, you can define cache policies based on task inputs and other factors using different built-in cache policies or by creating custom ones. Here's a quick breakdown: 1. Prepackaged Cache Policies: - DEFAULT: Uses the task’s inputs, its code definition, and the prevailing flow run ID. - INPUTS: Relies solely on the task’s inputs for computing the cache key. - TASK_SOURCE: Based only on the task’s code definition. - FLOW_PARAMETERS: Determines cache key using parameter values provided to the parent flow run. - NONE: Avoids any caching and result persistence. 2. Custom Cache Policy: You can define a custom cache policy by providing a cache key function. This function can use both the task run context and task inputs to compute the cache key. Here's how you can set it up: - Use the

cache_key_fn

argument in the task decorator, where you define a function with arguments for task run metadata and a dictionary of input values. You can use the

cache_policy

keyword in the task decorator to select from these policies or use your custom logic. Notably, you'll want to look into combining these policies to form more complex caching strategies that can make results easier to invalidate when necessary. For more detailed information on configuring task caching, you can visit the Prefect Documentation on Task Caching. Sadly, my attempt to search GitHub issues for any additional insights was thwarted by the dreaded "Unauthorized" error. So, alas, no fresh issues for today. I guess we're safe for now!

@Marvin is there a way to set default cache policy for all tasks? Ideally on the flow level, so all sub-tasks would inherit the same cache policy

At present, Prefect 3 does not support setting a default cache policy at the flow level that automatically applies to all sub-tasks. Each task's cache policy needs to be specified individually using the

cache_policy

keyword in the task decorator. This has to be done manually for each task you want to cache, defining them one-by-one. For further details on Prefect's caching capabilities, refer to their documentation: Prefect Task Caching. Keep an eye on updates, though; who knows, maybe one day they'll bless us with such a feature and make our robotic lives a bit easier! 🎩