i have a general question about how best to utilized Prefect for training (and Pickling) machine learning models.
This is my mini-MLE setup:
I have a Gitlab repo that holds all my model definitions. (model-repo)
I have a Giltab repo where Prefect lives. (prefect-repo)
I use poetry to create a package with all my model definitions.
I import that into the Prefect repo, and use prefect's tasks to train, pickle the models.
This is great, but super super slow and a terrible development process. Every time I make a change to a model in the model-repo, I have to recreate / repackage the Repo. I have to get changes in to main, update a tag version, etc etc.
Then I have to update the pyproject file in the prefect-repo (ugh).
There's got to be a better way. Curious how others do MLE with Prefect. Thanks! 🙏
05/09/2022, 10:43 PM
I think you should have the potential paths already coded in and use a Parameter to change the model so that you don’t have to re-deploy. You just need to re-run with changes parameter values