Hi - I need a possibility to run multiple snowflake queries in one session and sometimes in a transaction
I can probably easily add this - add parameter in SnowflakeQuery class:
• transaction: bool = false
• change query to queries that takes list of strings with queries - but this is not good for backward compatibility... I can maybe change that query is str or List[str]. Or maybe add another parameter: queries that will be taken into account when query is None.. or something
But please let me know if this makes sense for you and if it has a chance to me merged when finished
a
Anna Geller
05/30/2022, 5:31 PM
Good question - to avoid breaking things, how about a separate task for that purpose? Running multiple queries in a single transaction seems like a very different use case than the current tasks and might as well be designed as a separate task.
m
Marcin Grzybowski
05/30/2022, 5:32 PM
hmm, I have just found that there is SnowflakeQueriesFromFile that executes multiple queries...
but I have to check if it supports transactions, maybe it will be better to change this class
a
Anna Geller
05/30/2022, 5:33 PM
Based on your introduction, it looks like you are just getting started with Prefect - perhaps using Prefect 2.0 makes more sense for your PoC? given that we already have a snowflake collection for Prefect 2.0, you could contribute that directly there
m
Marcin Grzybowski
05/30/2022, 5:38 PM
yeah, I will ask my manager tomorrow if we can switch with our PoC to 2.0
it's not advanced so it should be quite fast
I will have to check a few things before, if for example prefect2.0 flow can be run as standalone python file (we will have in a few places a need to put that kind of script in a docker container and orchestrate container without the rest of infrastructure )
a
Anna Geller
05/30/2022, 5:41 PM
there is a DockerFlowRunner and KubernetesFlowRunner for that purpose - this page references various other resources that may help