Hi all I m working on some custom prefect serializers for pa

Question

Hi all I m working on some custom prefect serializers for pandas df results I have something that seems to work but I don t think it is very robust requires creating a local file ```class ParquetSerializer Serializer def serialize self value pd DataFrame gt bytes transform a Python object into bytes tmp filename = str time ns + parquet value to parquet path=tmp filename index=False with open tmp filename rb as in file df bytes = in file read Path tmp filename unlink return df bytes def deserialize self value bytes gt pd DataFrame recover a Python object from bytes df bytes io = BytesIO value df = pd read parquet df bytes io return df``` Does anyone have thoughts about the above approach saving as a local file then reading the bytes from the file

Bob Colner · Answer

Seem like there should be a better way

Bob Colner · Answer

I m using a timestamp in the file name b c so parallel tasks don t interfere with each other

Jeremiah · Answer

I m not familiar enough with Python parquet tools to know if there s an in memory bytes representation that would avoid the write to file step Perhaps someone else will know

Bob Colner · Answer

thanks there is no method to return parquet bytes that I m aware of Do you think my approach is the next best option I m not a real programmer lol

Jeremiah · Answer

Don t tell anyone neither am I slightly smiling face yes I think there s nothing wrong with your approach even though I understand why you re looking to optimize it

Bob Colner · Answer

cool I give it a try for a bigger workload and see how it goes If it works I m can try to do a pull request

Bob Colner · Answer

improved version ```class ParquetSerializer Serializer def serialize self value pd DataFrame gt bytes transform a Python object into bytes with NamedTemporaryFile mode= w+b as tmp ref1 open tmp ref1 name rb as tmp ref2 value to parquet path=tmp ref1 index=False df bytes = tmp ref2 read return df bytes def deserialize self value bytes gt pd DataFrame recover a Python object from bytes df bytes io = BytesIO value df = pd read parquet df bytes io return df```

Brett Naul · Answer

do you have a specific reason for wanting to use parquet as the intermediate format I believe dask just uses cloudpickle for dataframes and I think it has special handling for dataframes that makes it pretty efficient seems unlikely that this file roundtrip would be faster

Bob Colner · Answer

checkout <https arrow apache org docs python dataset html>

Brett Naul · Answer

looks cool it doesn t seem like your parquet serializer would be able to take advantage of those features though I m sure there are use cases for this but I guess I would just suggest you benchmark it and see how performance compares to cloudpickle since that requires no work on your part to use slightly smiling face we pass pretty big dataframes around like this a couple GB w o issue anything bigger than that I assume you wouldn t use a result for anyway

Bob Colner · Answer

the parquet feather datasets are working for my with my serializers improved with from the above version Lots of other big data tools expect parquet files in this dir sturucture hive presto etc very common use case in my world anyway

Bob Colner · Answer

none of these tools would work with cloudpickles

Brett Naul · Answer

sure totally but you can t leverage those by passing around the actual bytes you d need to write them to a persistent not tmp path and pass around paths instead which is definitely something I would be interested in too +1

Bob Colner · Answer

yeah I think the confusion is I m abusing the results checkpoint feature I m able to get my task to upload a parquet feather file in a dir structure on s3 This let me use the data with lots of useful tools that can query over all files arrow dataset presto etc It s working for me man shrugging