Hey all! Has anyone here needed to "package" regular files with their flows? We can, of course, upload "file packages" somewhere else and download them each time a flow runs, but specifically for the

DbtShellTask

, I'm thinking it would be nice if a flow we register and store on s3 also came with its dbt project directory. Currently, I don't think it's possible and will store the "packaged files" on s3, but I also don't want to start that work if there's already a solution that I haven't seen. 😄

Hey @William Grim, I think this article went about it my cloning the repo with those files.

Okay, then that's essentially the same idea. We thought either s3 or github, and it looks like that's the general right idea for this.

Indeed, there are two main options: either package everything in a Docker container or add a step to your flow to clone the dbt repo. Depends a bit on what you want, but you can’t go wrong either way 😉

@Matthias we did put everything in a docker container, but since the agent box is "local" and kicks everything off right now, it causes problems. I think the cloning is what we'll do soon, after we get our agents working as kubernetes agents or something.

What's causing the problems?

no deps