https://prefect.io
Join Slack
Hi, I'm trying to upgrade from agents to workers and trying to deploy my flows using the prefect.yam...
# prefect-cloud
m
Hi, I'm trying to upgrade from agents to workers and trying to deploy my flows using the prefect.yaml method. My docker registry is on ECR, and I see the container being pulled and ran successfully but I get a FileNotFoundError because the Path is not being populated in my deployment. I have the path defined in the deployment as well as the pull step in the YAML but I'm having no luck with populating it. Would appreciate any guidance!
n
hi @Moxit are you trying to use 
build_from_flow
with a 
prefect.yaml
? with workers, you should no longer use that or have to worry about a 
path
for your deployment, only an 
entrypoint
that you set in your 
prefect.yaml
that's relative to the root of the place that your code lives
m
I set the entrypoint but it's not the same path as what's in the container. Ex. locally the entrypoint is 
flows/flows/directory/flow_file.py:flow_name
and inside docker it is 
flows/directory/flow_file.py:flow_name
with a path variable of 
opt/prefect/flows
.
I'm only using prefect.yaml not with build_from_flow
👍 1
n
gotcha, have you used 
set_working_directory
in your 
pull
step?
m
Copy code
pull:
- prefect.deployments.steps.set_working_directory:
    directory: /opt/prefect/flows
I have it set to this
n
are using your own Dockerfile or the autobuild one?
m
my own
n
can u share it by chance?
m
Copy code
WORKDIR /opt/prefect

COPY flows/flows/ /opt/prefect/flows
COPY flows/poetry.lock ./
COPY tools /opt/tools
COPY flows/pyproject.prod.toml ./
COPY flows/pyproject.dev.toml ./pyproject.toml

RUN poetry install --no-dev && rm -rf $POETRY_CACHE_DIR

# Runtime stage
FROM --platform=linux/amd64 prefecthq/prefect:2.13.0-python3.9
RUN apt-get update  && apt-get install -y libpq-dev
# Copying only necessary artifacts from builder stage
COPY --from=builder /opt/prefect /opt/prefect
COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /usr/local/lib/python3.9/site-packages /usr/local/lib/python3.9/site-packages
COPY --from=builder /opt/tools /opt/tools

WORKDIR /opt/prefect
I can share this much of where I'm performing the copy of local files to the WORKDIR
n
just to sanity check, is this the actual structure you have on the building machine? i.e. 
flows/flows
Copy code
COPY flows/flows/ /opt/prefect/flows
m
Yeah
👍 1
the flow I'm trying to deploy has the entrypoint 
flows/flows/directory/directory.py:flow_name
In the error message I see this from the worker btw:
Copy code
prefect.exceptions.ScriptError: Script at 'flows/flows/normalization/normalization.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
 > Running set_working_directory step...
n
hm how is the 
entrypoint
set in your 
prefect.yaml
? also if its helpful sometimes i like to set little debug pull step like this or 
ls -la
✅ 1
m
Copy code
deployments:
- name: test-worker
  version: null
  tags: []
  description: null
  schedule: {}
  flow_name: null
  entrypoint: flows/flows/normalization/normalization.py:normalization_flow
  parameters: {}
  work_pool:
    name: null
    work_queue_name: null
    job_variables:
      image: '{{ build_image.image }}'
Let me add that
n
my naive hunch is that you actually want
Copy code
WORKDIR /opt/prefect

COPY flows/flows/ /opt/prefect # or just .
instead of
Copy code
WORKDIR /opt/prefect

COPY flows/flows/ /opt/prefect/flows
but yeah I would wanna log out stuff to double check myself
m
The dockerfile works when we use those same images to run via agents, would the worker change that?
n
hmm i dont think it should
m
I have it working after I changed my docker file to 
COPY flows/flows/ /opt/prefect/flows/flows
Thanks for the help!
n
ah! i had it backwards 🙂
glad to hear you got it working, sure thing!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by