https://prefect.io logo
Join Slack
Powered by
Hello folks, I’m looking for a tool which could au...
# ask-community
h
Hello folks, I’m looking for a tool which could automate our data processing steps and pytorch model training steps in our machine learning pipeline. Is prefect a good tool for this task? I can see prefect is good at ETL, but I’m not sure if prefect has good integrations with pytorch distributed training. Any suggestions? Thanks
k
Prefect can run this the ML pipeline for you. You just need to wrap the code as tasks and I’ve seen users do it. You can also use it together with other tools like MLFlow as well for experiment tracking.
h
@Kevin Kho thanks! Another quick question, can I deploy prefect orion to docker/private k8s cluster and then I can submit and run workflows there? I didn’t see a document about this.
k
Yes. See this maybe for Kubernetes. For Docker, we don’t have an official one yet but you can check this
h
I’ll check out the docs later, many thanks!
Hi Kevin, does prefect support to config a particular S3's endpoint? We use a private object storage which is compatible with S3, but we have to use our internal endpoint for that
k
How does it work with boto3? You give an address right?
Try passing it in the 
client_options
of S3 Storage ?
h
It works with boto3
I tried to use prefect storage create, but no client options available
Do you have an example about how to config S3 within python code? prefect cli doesn’t provide configuration of client_options
Seems that prefect.storage.S3 is not available for prefect2.0 orion
k
Oh that’s true. This is for Prefect 1. I don’t think S3 storage in 2.0 supports that
@Marvin open “Orion: S3 Storage does not support custom endpoints”
I guess you would need to use Prefect 1.0 to support that
m
h
Okay, thanks. I’ll try out prefect1
20 Views
Open in Slack
PreviousNext
Powered by