Hrm trying to make a Prod vs Test project with separate CI C Prefect Community #ask-community

Hrm, trying to make a Prod vs Test project, with s...

matta

07/31/2021, 2:58 AM

Hrm, trying to make a Prod vs Test project, with separate CI/CD pipelines. Ideally, people won't have to think about anything but making a feature branch - but I'm using Git storage, so for now I do have to tell it where the feature branch is. It's not THAT big of a deal, but is there any way to set it up so that it'll "know" to tell the storage to look in the feature branch that it's on? Thanks!

matta

07/31/2021, 2:59 AM

Maybe some kind of Env variable in the CI/CD? Does the code get "run" at all when you register with the CLI?

matta

07/31/2021, 3:02 AM

this is my Github Action for the Tester project right now:

Copy code

name: Feature CI

# Controls when the action will run. 
on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ feature/* ]
  pull_request:
    branches: [ feature/* ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    env:
      PREFECT__CLOUD__API_KEY: ${{ secrets.PREFECT_KEY }}
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: '3.8'
      - name: upgrade pip 
        shell: bash -l {0}
        run: pip install --upgrade pip
      - name: install prefect
        shell: bash -l {0}
        run: pip install "prefect[google]"
      - name: Register flows
        shell: bash -l {0}
        # Flows won't be Scheduled unless they're committed to Main
        run: prefect register --no-schedule --project tester -p flows/

Jeremy Phelps

07/31/2021, 7:44 AM

I had a similar problem, except I used Docker storage, so flows had to be told what their Docker image was (it differs between prod and staging). My solution was to create a subclass of Prefect's Flow class that specifies which image to use in the constructor:

Copy code

class MyFlow(prefect.Flow):
    def __init__(self, name, *args, **kwargs):
        super().__init__(self, name, *args, storage=Docker(registry_url='...', image_name='...'), **kwargs)
        self.run_config=UniversalRun(labels=staging_or_prod())
        self.executor=DaskExecutor(...)

👍 1

6 Views

Open in Slack

Previous Next