Hi everyone,
Has anyone wriiten methods to cache `...
# ask-communityd
Deceivious
04/27/2023, 9:21 AMHi everyone,
Has anyone wriiten methods to cache
Pandas Dataframepd.Dataframeread_csvDeceivious
04/27/2023, 12:28 PM Copy code
from prefect.serializers import Serializer, D
import pandas as pd
from io import BytesIO
class PandasSerializer(Serializer):
__dispatch_key__ = "pandas_serializer"
def dumps(self, obj: D) -> bytes:
return obj.to_csv(index=False).encode("utf8")
def loads(self, obj: D) -> pd.DataFrame:
return pd.read_csv(BytesIO(obj))Deceivious
04/27/2023, 12:29 PMEnded up writing one myself. Sharing it in case, some one wants to use it / test it / comment on it.
Deceivious
04/27/2023, 12:29 PMNot sure what or why dispatch key is required.
a
Avinash Santhanagopalan
04/27/2023, 2:33 PMWhy don’t you return the df in a task and cache the task?
d
Deceivious
04/27/2023, 2:34 PMI want the storage to have csv
👍 1
Deceivious
04/27/2023, 7:47 PM@Avinash Santhanagopalan also i dont think you can simply cache a pandas dataframe.
a
Avinash Santhanagopalan
04/27/2023, 8:55 PMI think you can cache the result of any task using cache_key_fn. So I think if you use that you can reuse the results of this data frame.
9 Views