Hi! I am working on setting up a simple CI/CD proc...
# prefect-community
k
Hi! I am working on setting up a simple CI/CD process that registers new flows each time a push is made to main. My folder structure is inspired by the gitlab data team so at the root of my project i have a folder orchestrate. Within that folder i have a folder tasks - within this there are subdirectories that contain custom modules that store tasks we use throughout our flows. While registering flows via the GitHub action, the Prefect CI is complaining about not being able to find a module in the tasks folder. When I run this locally, I do not get any issues. I am pretty sure it is related to my PYTHONPATH setup within the GitHub flow but I have not had any luck troubleshooting. Has anyone out there run into a similar issue?
1
k
Yes, the fix for this is mentioned here One straightforward way for you to fix this is either to add the module to PYTHONPATH or add the following code to the beginning of the script:
Copy code
import sys
sys.path.append("path_to_your_module")
k
What script am I adding this to? Here is my current job:
Copy code
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.7'
          cache: 'pip'

      - name: Install Prefect
        run: |
          cd $GITHUB_WORKSPACE
          pip install -r requirements.txt
          prefect register -p ./orchestrate/flows/authentics/database_loader.py --project $PREFECT_PROJECT
I'm assuming I'd run that script prior to registering
k
That would be inside the Python file, but I think you are right that you can just add to PYTHONPATH. Can’t you add it after installing the requirements.txt?
Like this?
Copy code
export PYTHONPATH="${PYTHONPATH}:/my/other/path"
and then the import will resolve?
k
I've tried that but the documentation I have found related to that within GitHub workflows is not very clear. I may shift and move towards using a python script to register instead of the CLI
a
we have an entire Discourse tag for that https://discourse.prefect.io/tag/ci-cd including a blog post from Jean for Prefect 1.0 https://www.prefect.io/guide/blog/deploying-prefect-flows-with-github-actions/
k
thanks!
Just in case anyone stumbles onto this thread in the future. I was able to successfully register my flows using GitHub actions by setting the PYTHONPATH variable like @Kevin Kho mentioned. It's just done a bit differently than other platforms.
Copy code
- name: Setup Python Env
        run: |
          echo "PYTHONPATH=$pythonLocation:$GITHUB_WORKSPACE/orchestrate" >> $GITHUB_ENV
      - name: Install Prefect
        run: |
          pip install -r requirements.txt
          prefect register -p orchestrate/flows/authentics/database_loader.py --project $PREFECT_PROJECT
k
Ah I see
177 Views