https://prefect.io logo
Title
w

William Grim

02/22/2022, 10:52 PM
Hey all! Has anyone here needed to "package" regular files with their flows? We can, of course, upload "file packages" somewhere else and download them each time a flow runs, but specifically for the
DbtShellTask
, I'm thinking it would be nice if a flow we register and store on s3 also came with its dbt project directory. Currently, I don't think it's possible and will store the "packaged files" on s3, but I also don't want to start that work if there's already a solution that I haven't seen. 😄
k

Kevin Kho

02/22/2022, 10:55 PM
Hey @William Grim, I think this article went about it my cloning the repo with those files.
But yes I guess it’s either packaging in a container or downloading. You can copy individual files using the Docker Storage interface too. I think you know all the options.
w

William Grim

02/22/2022, 10:58 PM
Okay, then that's essentially the same idea. We thought either s3 or github, and it looks like that's the general right idea for this.
I really appreciate the help.
m

Matthias

02/23/2022, 7:56 AM
Indeed, there are two main options: either package everything in a Docker container or add a step to your flow to clone the dbt repo. Depends a bit on what you want, but you can’t go wrong either way 😉
w

William Grim

02/23/2022, 5:15 PM
@Matthias we did put everything in a docker container, but since the agent box is "local" and kicks everything off right now, it causes problems. I think the cloning is what we'll do soon, after we get our agents working as kubernetes agents or something.
m

Matthias

02/23/2022, 6:34 PM
What's causing the problems?
w

William Grim

02/23/2022, 6:35 PM
no deps
That'll be what we fix using DockerStorage or something, if we can.
Easiest way to talk about it is mention the context of this knowledge: https://prefect-community.slack.com/archives/CL09KU1K7/p1645216750001739