https://prefect.io logo
#prefect-community
Title
# prefect-community
i

Iuliia Volkova

08/19/2022, 10:19 AM
Prefect 2 question. Remote storage. I'm trying to build & deploy pipeline according to documentation https://docs.prefect.io/concepts/deployments/#build-the-deployment. I run command as explain in doc. And I see that Prefect put in bucket WHOLE folder that I'm in (whole workdir) - take a look at the screen. If I will run command from my $HOME Path it will upload whole HOME path in remote bucket? If think.. by logic, it should upload only flow, or flow dir - path that defined in command, but not all environment around. Should I open issue for that? or it there is any idea under that behavior?
1
o

Oscar Björhn

08/19/2022, 10:27 AM
Hey, You're right, it does upload everything in the folder you're in (and below), unless you have a .prefectignore file where you've specified patterns to ignore. See the section about prefectignore files here: https://docs.prefect.io/concepts/deployments/#build-the-deployment
i

Iuliia Volkova

08/19/2022, 10:34 AM
but maybe better to do at least as in Docker - like possibility to provide context in command?
that to build
o

Oscar Björhn

08/19/2022, 10:36 AM
I agree. 🙂 Our solution right now is to upload more files than we actually need, so each flow has their own copy of.. every flow and related module, in their own storage paths. Not optimal, wastes space and bandwidth, but it works okay for now. I'm expecting the build command to be improved over time.
❤️ 3
🙏 2
💯 1
a

Anna Geller

08/19/2022, 11:13 AM
supporting flow code in Docker image is on immediate roadmap
🚀 2
j

Joao Moniz

08/19/2022, 1:42 PM
@Oscar Björhn just to better understand, basically you are creating a S3 Storage block for each one of your flows, is that right? I am having some issues with the deployment similar to the one Iuliia reported, looking for alternatives 🙂
o

Oscar Björhn

08/19/2022, 2:10 PM
Yeah, exactly. They all have the same base storage path but different subfolders, that way a developer working on flow_one won't overwrite the py files related to flow_two
👏 2
smart 3
🙌 1
j

Joao Moniz

08/19/2022, 2:13 PM
Thanks Oscar, this seems to solve the problem, that's a good hacky solution
o

Oscar Björhn

08/19/2022, 2:16 PM
Nice! 🙂
4 Views