Billy McMonagle
01/09/2024, 8:41 PMprefect.exceptions.ScriptError: Script at 'main.py' encountered an exception: ModuleNotFoundError("No module named '<flow dependency>'")
An exception occurred.
Billy McMonagle
01/09/2024, 8:43 PMprefect.yaml
file as follows:
name: data-eng-pipelines/geocode
prefect-version: 2.14.5
build: null
push: null
pull:
- prefect.deployments.steps.set_working_directory:
directory: /opt/prefect/
deployments:
- name: geocode-deployment
version: null
tags: []
description: null
schedule: {}
flow_name: null
entrypoint: main.py:main
parameters: {}
work_pool:
name: '{{ prefect.variables.workPool }}'
work_queue_name: null
job_variables:
image: '{{ prefect.variables.image }}:sample-{{ prefect.variables.tag }}'
I strongly wish to separate CI/CD from dependency install, if possible, because the goal of this project is to have various flows which have their own separately managed dependencies. Are there any recommended solutions?Kevin Grismore
01/09/2024, 8:44 PMBilly McMonagle
01/09/2024, 8:45 PMAlexander Azzam
01/09/2024, 8:56 PMBilly McMonagle
01/09/2024, 9:03 PMIan Thomas
01/10/2024, 9:19 PM- run:
name: Deploy the Prefect Flow
command: |
export ENVIRONMENT='PREFECT_DEPLOY'
prefect --no-prompt deploy --name <<parameters.environment>>
Then, in my main.py
where the deploy does a partial against the function in the entry point to grab the metadata off the flow decorator, (pulling in all the imports) I wrap the module imports unnecessary to the deploy within a conditional so they're skipped only during deploy.
if not getenv(
'ENVIRONMENT'
) == 'PREFECT_DEPLOY':
from dependent_module.app import main
@flow(
name='best-flow-evar'
)
def run_flow():
main()
if __name__ == "__main__":
run_flow()