Here is general question that I am really wondering about. What is the benefit of using Prefect and ECR Fargate together to run a machine learning end to end project while you can use SageMaker pipelines and parallelize your work? I like using Prefect and want to understand the benefits that I am not aware of. Thanks.

Hey @Payam K, I have not used Sagemaker in depth myself but you may find it easier if you are using their pre-built machine learning images whereas I guess in Prefect and ECR, you may have to supply your own. And from my understanding, if you use their pre-defined images, you get kind of a one-click deployment. I would say Prefect would be more help if there are intermediate steps there that would want observability around. What if you want to log your experiment tracking artifacts to somewhere like MLFlow? What if you want to persist batch predictions somewhere else? I think that is where Prefect can fit in. I would also say that SageMaker fight be overkill for certain ETL jobs. It may be unnecessarily expensive, whereas in Prefect, you can control your compute. If you have different pipelines that need varying compute, then orchestrating with Prefect lets you consolidate them in one place

Thank you for explanation.