Georgi Yanev

09/04/2022, 9:11 PM
Hey there, I I am reading and playing with prefect v2 for a while. I already have some flows working perfectly locally using local storage and dask, but now I have to move everything to use docker in preparation for production. We have minio server and I want to utilize it as a storage form my flow code. And here comes my struggle and question. Should I have separate s3 block for every flow code to use in the deployments? If so how should I reorganize my project structure, because currently all my flows are leaving in the same folder and it will be strange to have the same files deployed in multiple blocks.

Ryan Peden

09/05/2022, 1:08 AM
Hi Georgi, you don't need a separate S3/minio bucket for each deployment. If you'd like, you can use a single bucket and specify a different folder for each deployment. When you create a deployment, it will upload your flow code along with anything else in its directory and any subdirectories - except for anything include in your .prefectignore file. If you want to avoid uploading all your flows at once, you might consider putting each flow in its own subdirectory. If you'd like to test how it all works in practice when using MinIO, I've put together a GitHub repository with a Docker Compose file and a readme that will let you spin up various Prefect services and MinIO to get a feel for how they work together.

Georgi Yanev

09/05/2022, 7:40 AM
Thanks, I've tested using paths and this make everything much easier to manage. I already have a working docker compose file, but for sure I'll have a good look at your repository for inspirations and as a double check of my solution.

Vaikath Job

09/26/2022, 9:35 PM
@Georgi Yanev Wanted to ask a question about LocalFileStorage. Did you have to specify separate storage objects for each folder when using Local File Storage? I have been able to use relative paths for S3 but cannot figure out the same for LocalFileStorage and thought you might have run into this problem.