Does anyone know if Prefect is planning to add support for Snowpark Container Services, Snowflake's upcoming compute service? It's currently in preview, I think they're fairly close to a general release. If not, I'm considering looking into writing my own worker during the summer holidays.. I could hold off if it's already being worked on though, or perhaps some collaboration is possible. Getting really fed up with Azure Container Instances. 😅

Hi, just our of curiosity, what is so good about snowpark container services in your opinion? I had only heard negative speculations like how they use their own python version so there could be dependency issues, slow to release etc.

I think you might be mixing up Snowpark for Python and Snowpark Container Services! Snowpark Container Services is "just" a way to run docker-compliant containers on Snowflake's CPU, with their governance/security model etc. ACI is really unstable and has a poor API, getting logs out is annoying, and so on. Kubernetes was too much effort for us. Snowflake's stuff is generally stable so I expect their container services to deliver better quality, looks like it has the simplicity of ACI. Snowpark for Python lets you run Python code directly on their compute warehouses. We've been using it in production with great results so far, it's improved a lot over time. When it first released we couldn't use it for our purposes but we gave it another shot a few months later. We run a pretty heavy pipeline that reads huge json files and "censors" any sensitive data without otherwise transformering the files and running it on a week of data used to take something like 4 hours when we ran it outside of Snowpark, while we can do the same thing in 5-10 minutes in Snowpark. I don't know about the Python version being different, first I hear of it. Are you perhaps referring to the fact that they have a private conda registry for packages etc so you can't install things from pip? If that's what you mean, I think they do it for security reasons, so they can vet the packages/versions. Nowadays they have good support for uploading your own packages in various ways. We have 2 or 3 that we've only been able to find in pip and we can run them in Snowpark with no problems. This works even for dbt python models and that sort of thing.

Thank you, that's very informative. Yes, I was mixing up snowpark for python with snowpark container services, I had lumped them together as snowflake compute. We were debating on using Snowflake compute at one point but decided to go with AWS ECS instead. Glad to hear though that you have had great experience with both.

Probably the right decision considering Snowflake's compute is still in preview, I guess ECS is pretty stable. As long as it's better than ACI it can't be too bad. 😄 I don't have that much experience with snowpark container services yet, just getting enrolled into the preview program, so I can't vouch for it. But snowpark for python has been awesome!

@Oscar Björhn We are using ECS Push work pool. Works great most of the time. Still the connection is sometimes flaky, meaning ECS Tasks fails to start and communicate state back to Prefect’s backend. I agree it would be very interesting to be able to run Prefect on Snowflake’s compute. I’m just thinking what would be the price per minute compared to same sized instance on ECS.

Yeah, it seems like it'll be slightly more expensive than ACI but not by much. Not sure how those prices compare to ECS. It looks like the cheapest compute node costs about 1/7 of the smallest-size warehouse (XS).