https://prefect.io logo
Title
b

Bryan Whiting

02/16/2023, 10:00 PM
How does prefect cache? • I’m looking at Luigi, which just saves out artifacts as I’ve determined (pickle, joblib, csv, etc.). I want similar behavior in Prefect because I have computationally intensive intermediate tasks that i’d like use for analysis. • In Luigi, i can delete the artifact to re-start the task. In prefect can I clear the cache of a particular task?
r

redsquare

02/16/2023, 10:36 PM
Prefect caches task results in memory during a flow run, then to the path in PREFECT_LOCAL_STORAGE_PATH. You can of course clear the relevent entry by cache key
:thank-you: 2
🙌 1
b

Bryan Whiting

02/16/2023, 10:36 PM
what caching thing does it use? pickle?
b

Bryan Whiting

02/16/2023, 10:38 PM
You can define a task that is cached based on its inputs by using the Prefect task_input_hash. This is a task cache key implementation that hashes all inputs to the task using a JSON or cloudpickle serializer. If the task inputs do not change, the cached results are used rather than running the task until the cache expires.
thanks!