hi all trying to schedule ML predictions using prefect this

Question

hi all trying to schedule ML predictions using prefect this involves loading a pre trained pickled ML model refitting and saving new model daily The very first model can be saved either using a pickle joblib backend I know how to write manually the code for reading writing the model locally to s3 but has anyone done this using `LocalResult` and `S3Result`

Andreas Tsangarides · Answer

MLPIckleSerialzer py

Andreas Tsangarides · Answer

``` File Users tsangis Projects uk prefect flows venv lib python3 8 site packages prefect engine task runner py line 876 in get task run state value = prefect utilities executors run task with timeout File Users tsangis Projects uk prefect flows venv lib python3 8 site packages prefect utilities executors py line 454 in run task with timeout return task run args kwargs type ignore File Users tsangis Projects uk prefect flows src flows price predictions day ahead hourly tasks py line 80 in update model model = latest ml model read latest ml model loc value File Users tsangis Projects uk prefect flows venv lib python3 8 site packages prefect engine results local result py line 86 in read new value = self serializer deserialize value File Users tsangis Projects uk prefect flows src common results serializers py line 83 in deserialize return pickle load value TypeError file must have read and readline attributes```

Anna Geller · Answer

Good question have you heard about targets in Prefect This seems like a great fit for your use case You could template the target based on the date task name etc and simply returning your model would pickle it into the specified location Here are more details

Andreas Tsangarides · Answer

Hey Anna hope you are well yeah already using those and that s why I want to make use of the result objects it s the serializer I am having trouble with I think if the read method was missing from the local result it would work ``` local result py with open os path join self dir location rb as f value = f read ```

Anna Geller · Answer

Can you use the default <https github com PrefectHQ prefect blob 01705d38b9819a8e38d2696ee56999bc7a3faf89 src prefect engine serializers py L60|PickleSerializer> It uses cloudpickle which is better than pickle because it can serialize a function or class by value whereas pickle can only serialize it by reference

Andreas Tsangarides · Answer

in any case someones faces this It was the read method just need to convert bytes back to a buffer ```class MLPickleSerializer Serializer custom serializer for saving retrieving Ml models def deserialize self value bytes gt Any recover a Python object from bytes return pickle load io BytesIO value ```

Anna Geller · Answer

I m not sure whether this is needed and when LMK if you have any specific question about that Usually the default PickleSerializer works well to pickle arbitrary python objects incl ML models

Marco Barbero Mota · Answer

Following on this would this approach work for dictionaries

Marco Barbero Mota · Answer

I have found that the default pickle serializer does save ` pkl` files that cant be read when attempting using the module `pickle` or `pandas read pickle`

Andreas Tsangarides · Answer

hey Marco Migrating to Prefect 2 and using `Block`s for this kind of thing now