Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi guys, I’m planning to run Prefect Server on AKS. However, in order to execute our pipelines, the flow needs to read queries from external .sql files. What is the best way to store these sql files and how should Prefect access them? Appreciate any advice 

Hi Scarlett, is it the tasks that need access to the files?

I wonder if the easiest way would be to stick them in a storage container, provide your jobs with credentials to access that container, then you pull the file contents down during the flow runs as a first task?

I just skimmed the Flow Storage docs and there's nothing obvious about handling side-car style files

Though the Docker Storage docs <https://docs.prefect.io/orchestration/recipes/configuring_storage.html> do list
• special configuration files
As an example of why you would use it. You'd then build the files into the image alongside your flow

But if you're using another storage type and you don't want to do that, I'd use Blob Storage as a 'pre' task

So this would be part of the image that will be created in Azure Container Registry?

How are you currently registering your flows? But yes, it _could_ be created in any Docker registry

My general recommendation is one of:
• Create a dockerfile that has everything exactly how you'd like
• Use `Git` storage which clones the whole repo during flow load and pull the contents of the SQL files into constants
• Store the SQL files in `Git` or some other remote location and just pull them down explicitly in a task

<@U01CEUST9B5> Oooh I was unaware it cloned the *whole* repo, that is very fancy

Yeah with `GitHub` we do a single file clone but with `Git` we clone the whole thing which let's you use SSH/deploy keys for auth which people wanted. We still clean it all up before the flow runs but we want to explore letting people side-load files. It just gets complicated if you assume they're available but your task executes on a dask cluster or something.