https://prefect.io logo
Title
m

Matt Delacour

02/14/2023, 3:07 PM
👋 I don't understand how to create a deployment where the code is in a subdirectory in GitLab/Github
my_project.git
/src/
/src/dbt/
/src/dbt/Earthfile # Equivalent to Dockerfile
/src/dbt/my_prefect_job.py
/src/dbt/...
And my Earthfile is like
FROM prefecthq/prefect:2.7-python3.10
WORKDIR /code
...
COPY . .
And so
my_prefect_job.py
is under
/code/my_prefect_job.py
in my docker image and under
/src/dbt/
in my gitlab repository But once deployed, the run output the following
Downloading flow code from storage at 'src/dbt'

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1016, in get_code
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/code/prefect_job.py'
And so I don't understand why it cannot find my flow in gitlab under the right subdirectory nor the flow in the docker contain under the right path 🤷 Did not find any good example about github/gitlab subdir storage strategy. Thanks in advance 🙏
a

Aaron Gonzalez

02/14/2023, 3:59 PM
I'm curious what would happen if you
COPY . /opt/prefect/flows/
instead?
m

Matt Delacour

02/14/2023, 4:16 PM
I can try but I don't understand why it should work as I specified the WORKDIR in my Dockerfile
c

Christopher Boyd

02/14/2023, 7:58 PM
my guess would be because that’s not the entrypoint for the flow, and isn’t part of the pythonpath
Not sure if this is relevant, or oversight, but also:
FileNotFoundError: [Errno 2] No such file or directory: '/code/prefect_job.py'
has a different file name than what you reference:
/src/dbt/my_prefect_job.py
:plus-one: 1
m

Matt Delacour

02/14/2023, 9:47 PM
What is weird is that
/code/prefect_job.py
exists in my docker image while
/src/dbt/my_prefect_job.py
exists in my repository And I don't know for which source (docker image or repository) this FileNotFoundError refers to
I removed the WORKDIR in my docker image and I still get the same error
FileNotFoundError: [Errno 2] No such file or directory: '/code/prefect_job.py'
I cannot find any mention of
/code/prefect_job.py
in my repo. This is really weird
I got it working by not using any subfolder ... But it's not great, this is like forcing me to have one repo per project and not the flexibility to have sub projects within a repo 😢
c

Christopher Boyd

02/17/2023, 1:01 AM
You should have
path
and
entrypoint
- path is the relative path of your remote storage, and entrypoint is the file/flow defined. https://github.com/PrefectHQ/prefect/blob/main/src/prefect/deployments.py#L240 So i’d expect something like
path=/src/dbt/
and
entrypoint=my_prefect_job.py:<flowname>
To be fair though, you are right, there isn’t good documentation on this at the moment. It’s on my plate to add
b

Braun Reyes

04/21/2023, 5:48 PM
Not sure if this helps but I had that same issue here and solved it by making sure the github relative path matches the entrypoint path.
prefect deployment build -a data-products/src/orchestration/main_orchestration_flow.py:main_orchestration_flow -sb github/data-flows-dev/data-products/src/orchestration --name base
m

Matt Delacour

04/30/2023, 8:00 PM
Yes I made it work with something similar 🙌