Hello We are trying to deploy changes from a monorepo that h Prefect Community #best-practices

Hello, We are trying to deploy changes from a mono...

Abhinav Chordia

06/02/2023, 4:34 PM

Hello, We are trying to deploy changes from a monorepo that have multiple projects where our code lives. The structure looks something like this:

Copy code

python_repo /
   projectA / 
      src/projectA/flows/...
      pyproject.toml
   projectB /
      src/projectB/...
      pyproject.toml

projectA depends on projectB and I’d like to deploy all live changes within the python_repo. What is the recommended approach for doing this?

✅ 1

Jamie Zieziula

06/02/2023, 4:54 PM

Hi @Abhinav Chordia - you can use Prefect projects for this! You can initialize your repo as a project by running

prefect project init

. You’ll define a single prefect.yaml & deployment.yaml file at the root of your repo where you can specify the needed configuration for all of your flows

Jamie Zieziula

06/02/2023, 4:56 PM

your pull step in the

prefect.yaml

would look something like this:

Copy code

pull:
  - prefect.projects.steps.git_clone_project:
      repository: <https://github.com/YourOrg/python_repo>
      branch: main
      access_token: github_pat_token

this will tell prefect to pull and load your flow code from this repo at runtime

Jamie Zieziula

06/02/2023, 4:58 PM

and your deployment.yaml would look something like this (this is a very simplified version)

Copy code

deployments:
  - name: Project A Flow 1
    version: null
    tags: []
    description: null
    schedule: {}
    flow_name: null
    entrypoint: projectA/flows/flow1.py:flow
    parameters: {}
    work_pool:
      name: work-pool-name
      job_variables: {}
  - name: Project B Flow 1
    version: null
    tags: []
    description: null
    schedule: {}
    flow_name: null
    entrypoint: projectB/flows/flow1.py:flow
    parameters: {}
    work_pool:
      name: work-pool-name
      job_variables: {}

Abhinav Chordia

06/02/2023, 4:58 PM

Will this allow me to launch a flow from src/projectA/flows and pull in the deps from projectB? Locally we have projectB as an editable dep so we always get the latest changes.

Jamie Zieziula

06/02/2023, 5:00 PM

hm I see - I think for that you’d want to build a custom docker image that contains all of the dependencies your flow code needs (including the projectB code)

4 Views

Open in Slack

Previous Next