[Looking for Sagemaker + prefect feedback] Hey everyone ! New to prefect slack, so let me know if this is not the right place to post. I'm an ML engineer, picking tooling for my company's research oriented ML stack. Standard stuff, we do extensive data ETL, build a dataset, and then train a bunch of models we want to compare.

prefect

seems to be the best fit, because it is not invasive, and enables you to take exiting well formatted python code and put into tasks and flow, with handy caching on task output. However, we heavily depend on Sagemaker for our compute, and use Sagemaker studio. So we also considered

airflow

, which does not fit our needs so well (they push to avoid data being passed between tasks, no caching), but has extensive integration with Sagemaker. So: does anyone here have feedback on using

prefect

with Sagemaker, and is there any plans to propose more integration in the future ? I want to push

prefect

to my leadership and need proof points that we won't run into integration issues two months from now

hi @Nathan Bigaud Koenigswarter !

is there any plans to propose more integration in the future

there are no specific plans around Sagemaker, but if you were interested, you could use `prefect-aws` as a starting place and/or contribute sagemaker specific integrations to

prefect-aws

we'd be happy to help any contribution along! otherwise feature requests are also welcome