https://prefect.io logo
#prefect-community
Title
# prefect-community
h

HaoboGu

05/24/2022, 2:27 AM
Hello folks, I’m looking for a tool which could automate our data processing steps and pytorch model training steps in our machine learning pipeline. Is prefect a good tool for this task? I can see prefect is good at ETL, but I’m not sure if prefect has good integrations with pytorch distributed training. Any suggestions? Thanks
k

Kevin Kho

05/24/2022, 2:37 AM
Prefect can run this the ML pipeline for you. You just need to wrap the code as tasks and I’ve seen users do it. You can also use it together with other tools like MLFlow as well for experiment tracking.
h

HaoboGu

05/24/2022, 3:31 AM
@Kevin Kho thanks! Another quick question, can I deploy prefect orion to docker/private k8s cluster and then I can submit and run workflows there? I didn’t see a document about this.
k

Kevin Kho

05/24/2022, 3:34 AM
Yes. See this maybe for Kubernetes. For Docker, we don’t have an official one yet but you can check this
h

HaoboGu

05/24/2022, 3:41 AM
I’ll check out the docs later, many thanks!
Hi Kevin, does prefect support to config a particular S3's endpoint? We use a private object storage which is compatible with S3, but we have to use our internal endpoint for that
k

Kevin Kho

05/24/2022, 6:22 AM
How does it work with boto3? You give an address right?
Try passing it in the
client_options
of S3 Storage ?
h

HaoboGu

05/24/2022, 6:22 AM
It works with boto3
I tried to use prefect storage create, but no client options available
Do you have an example about how to config S3 within python code? prefect cli doesn’t provide configuration of client_options
Seems that prefect.storage.S3 is not available for prefect2.0 orion
k

Kevin Kho

05/24/2022, 6:36 AM
Oh that’s true. This is for Prefect 1. I don’t think S3 storage in 2.0 supports that
@Marvin open “Orion: S3 Storage does not support custom endpoints”
I guess you would need to use Prefect 1.0 to support that
h

HaoboGu

05/24/2022, 6:38 AM
Okay, thanks. I’ll try out prefect1
2 Views