[Looking for Sagemaker + prefect feedback] Hey everyone ! New to prefect slack, so let me know if this is not the right place to post (I've also posted in the community channel). I'm an ML engineer, picking tooling for my company's research oriented ML stack. Standard stuff, we do extensive data ETL, build a dataset, and then train a bunch of models we want to compare.

prefect

seems to be the best fit, because it is not invasive, and enables you to take exiting well formatted python code and put into tasks and flow, with handy caching on task output. However, we heavily depend on Sagemaker for our compute, and use Sagemaker studio. So we also considered

airflow

, which does not fit our needs so well (they push to avoid data being passed between tasks, no caching), but has extensive integration with Sagemaker. So: does anyone here have feedback on using

prefect

with Sagemaker, and is there any plans to propose more integration in the future ? I want to push

prefect

to my leadership and need proof points that we won't run into integration issues two months from now