https://prefect.io logo
Join Slack
Powered by
Hi, I'm trying to upgrade from agents to workers a...
# prefect-cloud
m
Hi, I'm trying to upgrade from agents to workers and trying to deploy my flows using the prefect.yaml method. My docker registry is on ECR, and I see the container being pulled and ran successfully but I get a FileNotFoundError because the Path is not being populated in my deployment. I have the path defined in the deployment as well as the pull step in the YAML but I'm having no luck with populating it. Would appreciate any guidance!
n
hi @Moxit are you trying to use 
build_from_flow
with a 
prefect.yaml
? with workers, you should no longer use that or have to worry about a 
path
for your deployment, only an 
entrypoint
that you set in your 
prefect.yaml
that's relative to the root of the place that your code lives
m
I set the entrypoint but it's not the same path as what's in the container. Ex. locally the entrypoint is 
flows/flows/directory/flow_file.py:flow_name
and inside docker it is 
flows/directory/flow_file.py:flow_name
with a path variable of 
opt/prefect/flows
.
I'm only using prefect.yaml not with build_from_flow
👍 1
n
gotcha, have you used 
set_working_directory
in your 
pull
step?
m
Copy code
pull:
- prefect.deployments.steps.set_working_directory:
    directory: /opt/prefect/flows
I have it set to this
n
are using your own Dockerfile or the autobuild one?
m
my own
n
can u share it by chance?
m
Copy code
WORKDIR /opt/prefect

COPY flows/flows/ /opt/prefect/flows
COPY flows/poetry.lock ./
COPY tools /opt/tools
COPY flows/pyproject.prod.toml ./
COPY flows/pyproject.dev.toml ./pyproject.toml

RUN poetry install --no-dev && rm -rf $POETRY_CACHE_DIR

# Runtime stage
FROM --platform=linux/amd64 prefecthq/prefect:2.13.0-python3.9
RUN apt-get update  && apt-get install -y libpq-dev
# Copying only necessary artifacts from builder stage
COPY --from=builder /opt/prefect /opt/prefect
COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /usr/local/lib/python3.9/site-packages /usr/local/lib/python3.9/site-packages
COPY --from=builder /opt/tools /opt/tools

WORKDIR /opt/prefect
I can share this much of where I'm performing the copy of local files to the WORKDIR
n
just to sanity check, is this the actual structure you have on the building machine? i.e. 
flows/flows
Copy code
COPY flows/flows/ /opt/prefect/flows
m
Yeah
👍 1
the flow I'm trying to deploy has the entrypoint 
flows/flows/directory/directory.py:flow_name
In the error message I see this from the worker btw:
Copy code
prefect.exceptions.ScriptError: Script at 'flows/flows/normalization/normalization.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
 > Running set_working_directory step...
n
hm how is the 
entrypoint
set in your 
prefect.yaml
? also if its helpful sometimes i like to set little debug pull step like this or 
ls -la
✅ 1
m
Copy code
deployments:
- name: test-worker
  version: null
  tags: []
  description: null
  schedule: {}
  flow_name: null
  entrypoint: flows/flows/normalization/normalization.py:normalization_flow
  parameters: {}
  work_pool:
    name: null
    work_queue_name: null
    job_variables:
      image: '{{ build_image.image }}'
Let me add that
n
my naive hunch is that you actually want
Copy code
WORKDIR /opt/prefect

COPY flows/flows/ /opt/prefect # or just .
instead of
Copy code
WORKDIR /opt/prefect

COPY flows/flows/ /opt/prefect/flows
but yeah I would wanna log out stuff to double check myself
m
The dockerfile works when we use those same images to run via agents, would the worker change that?
n
hmm i dont think it should
m
I have it working after I changed my docker file to 
COPY flows/flows/ /opt/prefect/flows/flows
Thanks for the help!
n
ah! i had it backwards 🙂
glad to hear you got it working, sure thing!
Open in Slack
PreviousNext
Powered by