Hi guys, I’m planning to run Prefect Server on AKS. However, in order to execute our pipelines, the flow needs to read queries from external .sql files. What is the best way to store these sql files and how should Prefect access them? Appreciate any advice

Hi Scarlett, is it the tasks that need access to the files?

Yes, the tasks need to access the files

I wonder if the easiest way would be to stick them in a storage container, provide your jobs with credentials to access that container, then you pull the file contents down during the flow runs as a first task?

So this would be part of the image that will be created in Azure Container Registry?

How are you currently registering your flows? But yes, it could be created in any Docker registry

My general recommendation is one of: • Create a dockerfile that has everything exactly how you'd like • Use

Git

storage which clones the whole repo during flow load and pull the contents of the SQL files into constants • Store the SQL files in

Git

or some other remote location and just pull them down explicitly in a task

@Zanie Oooh I was unaware it cloned the whole repo, that is very fancy

Yeah with

GitHub

we do a single file clone but with

Git

we clone the whole thing which let's you use SSH/deploy keys for auth which people wanted. We still clean it all up before the flow runs but we want to explore letting people side-load files. It just gets complicated if you assume they're available but your task executes on a dask cluster or something.