Currently when I deploy multiple flows I basically loop over Prefect Community #ask-community

Currently when I deploy multiple flows I basically loop over them and for each, build and apply the flow. Also found it on one of @Anna Geller’s posts/examples (https://github.com/anna-geller/dataflow-ops/blob/main/.github/workflows/main.yaml#L35). [I’ve put my version in the thread] This works fine but I was wondering if this is the preferred method? My concern is that I basically duplicate the storage-block (codebase) for each flow on the bucket. Is it possible to have a shared storage-block? It seems waste of deployment time to build the storage repeatedly. I understand that this course makes sense if you have different versions or not deploy all flows at the same time.

✅ 1

Joël Luijmes

12/15/2022, 2:52 PM

Copy code

release:
    stage: release
  
    parallel:
      matrix:
        - FLOW_PATH: src/flows/a.py
          STORAGE_PATH: a
        - FLOW_PATH: src/flows/b.py
          STORAGE_PATH: b
  
    interruptible: true
  
    script:
      - python -m venv .venv
      - source .venv/bin/activate
      - pip install poetry
      - poetry install
  
      - prefect deployment build "$FLOW_PATH:main"  --name "Production" --infra-block "kubernetes-job/default" --storage-block "gcs/flow-store/$STORAGE_PATH" --work-queue "kubernetes"
      - cat main-deployment.yaml
      - prefect deployment apply main-deployment.yaml

Anna Geller

12/15/2022, 4:18 PM

Very good question. Not sure whether you noticed but the way I implemented that in this demo was that I only upload code once using this maintenance flow, everything else has the skip upload flag to avoid any duplicated uploads When you leverage GitHub block instead of S3, this gets even easier since this way we always skip upload by default

Joël Luijmes

12/18/2022, 8:27 AM

Oe nice, thanks for the reply! I just quickly skimmed through it so missed that 🙊 But that is great, then I can just do that 🙂

Joël Luijmes

12/18/2022, 8:31 AM

Random: Also like the idea of the healthcheck flow. I’m going to add an additional step, to make a push step (statuscake) to alert me in case none of the flows are running 🙂

upvote 1

15 Views

Open in Slack

Previous Next