In the Snowflake Summit, they showed a glimpse of using Airflow with Snowflake Container Service as the backing infrastructure. As users of Snowflake, just want to point our that having the Snowflake Container Service as an Infrastructure option in Prefect would be a huge feature. Currently we had to use AWS ECS as the Infra for Prefect and needed to open a new AWS account. This was especially time consuming regarding security and compliance. Would love to know if this is something being talked about at Prefect.

this looks insane, if it would work as I expect it to… However, seems to me like most jobs that we’d want to run over Snowflake (even as containers) would be ones we could place as part of our dbt “data build” process, while Prefect would just be used to orchestrate things (DBT, workflow dependencies and interaction with external systems) in other words, I don’t think i would necessarily want to run ANY generic job using that infra, rather than k8s for example

@Tom Klein I just think that currently the "Infra" part of Prefect Cloud is a big defocus. I want something that just works and gets out of your way.

@Yaron Levi I agree, for the most part K8s is like that for us, the main issue we run into is when we need a burst of compute resources and scaling out (by parallelisation ) the execution. So far using Ray/Dask didn’t seem to work well for us. This is where (i think) Snowflake could shine - since it does it so well for SQL based transformations.

Yes, and also think that for many dev teams, things like K8s are completely foreign. For example, our team uses Render for compute.