I’m looking for a tool which could automate our data processing steps and pytorch model training steps in our machine learning pipeline. Is prefect a good tool for this task? I can see prefect is good at ETL, but I’m not sure if prefect has good integrations with pytorch distributed training. Any suggestions? Thanks
05/24/2022, 2:37 AM
Prefect can run this the ML pipeline for you. You just need to wrap the code as tasks and I’ve seen users do it. You can also use it together with other tools like MLFlow as well for experiment tracking.
05/24/2022, 3:31 AM
@Kevin Kho thanks! Another quick question, can I deploy prefect orion to docker/private k8s cluster and then I can submit and run workflows there? I didn’t see a document about this.
05/24/2022, 3:34 AM
Yes. See this maybe for Kubernetes. For Docker, we don’t have an official one yet but you can check this
05/24/2022, 3:41 AM
I’ll check out the docs later, many thanks!
Hi Kevin, does prefect support to config a particular S3's endpoint? We use a private object storage which is compatible with S3, but we have to use our internal endpoint for that
05/24/2022, 6:22 AM
How does it work with boto3? You give an address right?