Hello folks I m looking for a tool which could automate our

Question

Hello folks I m looking for a tool which could automate our data processing steps and pytorch model training steps in our machine learning pipeline Is prefect a good tool for this task I can see prefect is good at ETL but I m not sure if prefect has good integrations with pytorch distributed training Any suggestions Thanks

Kevin Kho · Answer

Prefect can run this the ML pipeline for you You just need to wrap the code as tasks and I ve seen users do it You can also use it together with other tools like MLFlow as well for experiment tracking

HaoboGu · Answer

< Kevin Kho> thanks Another quick question can I deploy prefect orion to docker private k8s cluster and then I can submit and run workflows there I didn t see a document about this

Kevin Kho · Answer

Yes See maybe for Kubernetes For Docker we don t have an official one yet but you can check

HaoboGu · Answer

I ll check out the docs later many thanks

HaoboGu · Answer

Hi Kevin does prefect support to config a particular S3 s endpoint We use a private object storage which is compatible with S3 but we have to use our internal endpoint for that

Kevin Kho · Answer

How does it work with boto3 You give an address right

Kevin Kho · Answer

Try passing it in the `client options` of <https github com PrefectHQ prefect blob 40e2e2bd1c1de5651bc7fb484c823e43cfc69877 src prefect storage s3 py L40|S3 Storage>

HaoboGu · Answer

It works with boto3

HaoboGu · Answer

I tried to use prefect storage create but no client options available

HaoboGu · Answer

Do you have an example about how to config S3 within python code prefect cli doesn t provide configuration of client options

HaoboGu · Answer

Seems that prefect storage S3 is not available for prefect2 0 orion

Kevin Kho · Answer

Oh that s true This is for Prefect 1 I don t think S3 storage in 2 0 supports <https orion docs prefect io api ref prefect blocks storage prefect blocks storage S3StorageBlock|that>

Kevin Kho · Answer

< Marvin> open Orion S3 Storage does not support custom endpoints

Kevin Kho · Answer

I guess you would need to use Prefect 1 0 to support that

Marvin · Answer

Marvin · Answer

HaoboGu · Answer

Okay thanks I ll try out prefect1