Hey all! Has anyone here needed to "package" regul...
# prefect-community
Hey all! Has anyone here needed to "package" regular files with their flows? We can, of course, upload "file packages" somewhere else and download them each time a flow runs, but specifically for the
, I'm thinking it would be nice if a flow we register and store on s3 also came with its dbt project directory. Currently, I don't think it's possible and will store the "packaged files" on s3, but I also don't want to start that work if there's already a solution that I haven't seen. 😄
Hey @William Grim, I think this article went about it my cloning the repo with those files.
But yes I guess it’s either packaging in a container or downloading. You can copy individual files using the Docker Storage interface too. I think you know all the options.
Okay, then that's essentially the same idea. We thought either s3 or github, and it looks like that's the general right idea for this.
I really appreciate the help.
Indeed, there are two main options: either package everything in a Docker container or add a step to your flow to clone the dbt repo. Depends a bit on what you want, but you can’t go wrong either way 😉
@Matthias we did put everything in a docker container, but since the agent box is "local" and kicks everything off right now, it causes problems. I think the cloning is what we'll do soon, after we get our agents working as kubernetes agents or something.
What's causing the problems?
no deps
That'll be what we fix using DockerStorage or something, if we can.
Easiest way to talk about it is mention the context of this knowledge: https://prefect-community.slack.com/archives/CL09KU1K7/p1645216750001739