https://prefect.io
Join Slack
Hrm, can you turn a Prefect Flow into a Snowflake External Function? <https://docs.snowflake.com/en/...
# ask-community
m
Hrm, can you turn a Prefect Flow into a Snowflake External Function? https://docs.snowflake.com/en/sql-reference/external-functions-introduction.html
d
Wow - cool idea. I would assume prefect server exposes and API to start a flow.
c
Yup, the Prefect API has a 
create_flow_run
route that allows you to trigger a flow run with configurable start time, parameters, context, and run name!
m
Awesome!
In case folks are interested: Looking for the best way to add ML model predictions to data we have in Snowflake and still keep as much as possible where dbt can "see" it and manage the tables and stuff.
Original plan was to regularly have Prefect run inference jobs on a target table, export a flat file to a Snowflake stage, then add that as a Data Source for dbt
Buuut then I saw this article, and wondered if it'd make sense to do this (except with Prefect instead of SageMaker): https://community.snowflake.com/s/article/How-To-Use-Amazon-SageMaker-model-with-data-in-Snowflake-directly-in-SQL
Which has the advantage of running automatically every time dbt rebuilds the tables
Although, I guess a downside vs "just run bulk jobs" is that it'd probably be less efficient compute-wise...
12 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by