Hey everyone!
I have a dataframe that is being passed between tasks in the flow and by using cache it is much faster and there is no need to persist it to disk.
My question is, is it possible to clear that big dataframe from cache completely after the flow is done and not by setting cache_expiration?
I read into refreshing the cache but that did not help since I do not repeat tasks.
I also tried with
del dataframe
gc.collect()
But it does not get released from cache
Any ideas?