Matthew Alhonte

02/16/2021, 12:31 AM
Hrm, can you turn a Prefect Flow into a Snowflake External Function? https://docs.snowflake.com/en/sql-reference/external-functions-introduction.html

David Clements

02/16/2021, 12:40 AM
Wow - cool idea. I would assume prefect server exposes and API to start a flow.

Chris White

02/16/2021, 1:33 AM
Yup, the Prefect API has a
route that allows you to trigger a flow run with configurable start time, parameters, context, and run name!


02/16/2021, 2:03 AM
In case folks are interested: Looking for the best way to add ML model predictions to data we have in Snowflake and still keep as much as possible where dbt can "see" it and manage the tables and stuff.
Original plan was to regularly have Prefect run inference jobs on a target table, export a flat file to a Snowflake stage, then add that as a Data Source for dbt
Buuut then I saw this article, and wondered if it'd make sense to do this (except with Prefect instead of SageMaker): https://community.snowflake.com/s/article/How-To-Use-Amazon-SageMaker-model-with-data-in-Snowflake-directly-in-SQL
Which has the advantage of running automatically every time dbt rebuilds the tables
Although, I guess a downside vs "just run bulk jobs" is that it'd probably be less efficient compute-wise...