Hi team I currently manage around 20 deployments each packag Prefect Community #ask-community

Hi team, I currently manage around 20 deployments,...

Fugo Takefusa

08/05/2025, 1:33 AM

Hi team, I currently manage around 20 deployments, each packaged as a separate Docker image. Every deployment uses a

helper

folder containing useful Python files. However, in our current folder structure, each deployment has its own duplicated copy of this

helper

folder. This setup is becoming difficult to maintain—whenever we need to update a file in

helper

, I have to manually apply the change across all 20 folders. I’m looking for a way to centralize the

helper

folder so that all deployments can reference a single shared version. Ideally, any updates made to the shared

helper

folder would automatically be reflected across all deployments. Does anyone have suggestions or best practices for implementing this kind of shared structure in a Docker-based environment? Here is a sample current structure:

Copy code

deployments/
├── deployment_1/
│   ├── flow.py
│   └── helpers/
│       ├── useful.py
│       └── useful2.py
├── deployment_2/
│   ├── flow_2.py
│   └── helpers/
│       └── useful.py
└── deployment_3/
    ├── flow_3.py
    └── helpers/
        ├─...

Joao Moniz

08/05/2025, 3:14 PM

I guess what you might be looking for is to keep the

helpers

folder centralized in one place in your project (common utils), and copy the folder to the docker image (as each deployment is packaged as a docker image). Smt like that:

Copy code

COPY /helpers ./helpers

Fugo Takefusa

08/05/2025, 8:36 PM

Thanks @Joao Moniz. Your solution would work. However, if I make a change in the helpers file, I need to re-deploy all existing deployments to make the changes available. I want to avoid that.

Joao Moniz

08/06/2025, 6:43 AM

Hmm it makes sense. Basically what we use is Github deployment to fetch the flow latest code, and a single package (that contains the common functions). so we have only 1 docker image that is shared among all deployments. Probably there are other solutions, try to ask @Marvin or maybe check if other deployment options could help you here.

Marvin

08/06/2025, 6:43 AM

✅ thought for 48.3 seconds

Marvin

08/06/2025, 6:44 AM

Based on the research and discussions, let me outline the key deployment strategies and options available in Prefect 3.x for your use case: 1. Current Approach (Git-based + Shared Docker Image) Your current setup is actually a good pattern! Using Git-based storage to fetch flow code while maintaining a single shared Docker image for common dependencies is an efficient approach. This provides: - Version control for flow code - Reduced storage overhead - Simplified dependency management - Easy updates to shared functions 2. Alternative Options: a) Remote Storage Options: - You can store flow code in cloud storage (S3, GCS, Azure Blob) using storage blocks - Useful when Git isn't preferred or for large auxiliary files - Link: https://docs-3.prefect.io/v3/how-to-guides/deployments/store-flow-code#remote-storage b) Docker-based Deployment Variations: - Build separate images per flow (more isolation, but higher overhead) - Use multi-stage builds to optimize image size - Use base images with common dependencies and extend as needed - Mount volumes for dynamic code/config updates c) Hybrid Approach: - Store core functions in the Docker image - Keep flow-specific code in Git/remote storage - Use Python packages for shared code (installable via pip) 3. Best Practices for Your Current Setup:

Copy code

yaml
# Example prefect.yaml structure
deployments:
  - name: example-deployment
    entrypoint: "path/to/flow.py:flow_fn"
    work_pool:
      name: your-pool
    pull:
      - prefect.deployments.steps.git_clone:
          repository: "<https://github.com/your/repo>"
          branch: main

The shared Docker image approach you're using is particularly effective when: - Multiple flows share common dependencies - You want to minimize storage and deployment overhead - Teams need to maintain consistent environments - Quick updates to flow logic are needed without rebuilding images One potential enhancement to consider: You could use Python packaging for your common functions and host them in a private PyPI repository or install directly from Git. This would make dependency management even more explicit and versioned. Would you like me to elaborate on any of these approaches or provide more specific implementation details for any particular strategy?

Fugo Takefusa

08/07/2025, 1:54 AM

Will consider changing the approach to use the github

6 Views

Open in Slack

Previous Next