a

    Alex Furrier

    1 year ago
    Is there a recommended file format for flat file storage created from a DataFrame? The data may have mixed data types including arrays stored as a value. In the past I've had trouble with complex data types using
    feather
    format and sometimes ran into errors with
    HDF
    as well
    m

    Mariia Kerimova

    1 year ago
    Hello Alex! Prefect is data format agnostic, and you can use any data types. I personally never used feather, but let's see if community has something to share.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Alex Furrier, Mariia is right that Prefect is data format agnostic. Mixed types are generally hard to deal with and I think that you will probably run into errors with
    feather
    because Apache Arrow is strongly typed. If you want to force this, you can turn that column into a binary blob and then use
    feather
    . When you load it and put it in pandas, by unpickling, I think it will work.
    d

    Dharhas Pothina

    1 year ago
    For tabular data Parquet is a very good option.