https://prefect.io logo
Title
k

Kevin

07/05/2022, 5:28 PM
Hi! I am working on setting up a simple CI/CD process that registers new flows each time a push is made to main. My folder structure is inspired by the gitlab data team so at the root of my project i have a folder orchestrate. Within that folder i have a folder tasks - within this there are subdirectories that contain custom modules that store tasks we use throughout our flows. While registering flows via the GitHub action, the Prefect CI is complaining about not being able to find a module in the tasks folder. When I run this locally, I do not get any issues. I am pretty sure it is related to my PYTHONPATH setup within the GitHub flow but I have not had any luck troubleshooting. Has anyone out there run into a similar issue?
1
k

Khuyen Tran

07/05/2022, 5:31 PM
Yes, the fix for this is mentioned here One straightforward way for you to fix this is either to add the module to PYTHONPATH or add the following code to the beginning of the script:
import sys
sys.path.append("path_to_your_module")
k

Kevin

07/05/2022, 5:35 PM
What script am I adding this to? Here is my current job:
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.7'
          cache: 'pip'

      - name: Install Prefect
        run: |
          cd $GITHUB_WORKSPACE
          pip install -r requirements.txt
          prefect register -p ./orchestrate/flows/authentics/database_loader.py --project $PREFECT_PROJECT
I'm assuming I'd run that script prior to registering
k

Kevin Kho

07/05/2022, 5:39 PM
That would be inside the Python file, but I think you are right that you can just add to PYTHONPATH. Can’t you add it after installing the requirements.txt?
Like this?
export PYTHONPATH="${PYTHONPATH}:/my/other/path"
and then the import will resolve?
k

Kevin

07/05/2022, 5:43 PM
I've tried that but the documentation I have found related to that within GitHub workflows is not very clear. I may shift and move towards using a python script to register instead of the CLI
a

Anna Geller

07/05/2022, 6:02 PM
we have an entire Discourse tag for that https://discourse.prefect.io/tag/ci-cd including a blog post from Jean for Prefect 1.0 https://www.prefect.io/guide/blog/deploying-prefect-flows-with-github-actions/
k

Kevin

07/05/2022, 6:06 PM
thanks!
Just in case anyone stumbles onto this thread in the future. I was able to successfully register my flows using GitHub actions by setting the PYTHONPATH variable like @Kevin Kho mentioned. It's just done a bit differently than other platforms.
- name: Setup Python Env
        run: |
          echo "PYTHONPATH=$pythonLocation:$GITHUB_WORKSPACE/orchestrate" >> $GITHUB_ENV
      - name: Install Prefect
        run: |
          pip install -r requirements.txt
          prefect register -p orchestrate/flows/authentics/database_loader.py --project $PREFECT_PROJECT
k

Kevin Kho

07/05/2022, 9:30 PM
Ah I see