How does prefect cache? - I’m looking at Luigi, which just saves out artifacts as I’ve determined (p...

Bryan Whiting

02/16/2023, 10:00 PM

How does prefect cache? • I’m looking at Luigi, which just saves out artifacts as I’ve determined (pickle, joblib, csv, etc.). I want similar behavior in Prefect because I have computationally intensive intermediate tasks that i’d like use for analysis. • In Luigi, i can delete the artifact to re-start the task. In prefect can I clear the cache of a particular task?

redsquare

02/16/2023, 10:36 PM

Prefect caches task results in memory during a flow run, then to the path in PREFECT_LOCAL_STORAGE_PATH. You can of course clear the relevent entry by cache key

🙌 1

🙏 2

Bryan Whiting

02/16/2023, 10:36 PM

what caching thing does it use? pickle?

redsquare

02/16/2023, 10:37 PM

https://docs.prefect.io/concepts/tasks/#caching

Bryan Whiting

02/16/2023, 10:38 PM

Copy code

You can define a task that is cached based on its inputs by using the Prefect task_input_hash. This is a task cache key implementation that hashes all inputs to the task using a JSON or cloudpickle serializer. If the task inputs do not change, the cached results are used rather than running the task until the cache expires.

Bryan Whiting

02/16/2023, 10:38 PM

thanks!

5 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.