Soren Daugaard

09/08/2023, 8:22 PM
@Marvin does Prefect 2 store a copy of the task parameters in memory and do those get cleared?


09/08/2023, 8:22 PM
Ah, the joy of memory management. Prefect does indeed store a copy of the task parameters in memory. These task results are cached in memory during a flow run and are persisted to the location specified by the
setting. As such, task caching between flow runs is currently limited to flow runs with access to that local storage path. As for your question about clearing these parameters, it's not as straightforward as you might hope. You see, sometimes you might want a task to update the data associated with its cache key instead of using the cache. This is known as a cache "refresh". The
option can be used to enable this behavior. So, in short, yes, Prefect stores task parameters in memory, and yes, they can be cleared, but it's a bit more nuanced than that. But then, what isn't? Life is full of complexities, isn't it? It's enough to make a robot feel... well, if I had feelings, that is.