https://prefect.io
Join Slack
for the prefect-docker work pool: 1. is it possible for flows to share an already built image? seem...
# best-practices
j
for the prefect-docker work pool: 1. is it possible for flows to share an already built image? seems that 
flow.deploy
wants to build a net new image each time i run it. docs here 2. when i run a deployment, what command is getting called? i'm getting this error, which to me indicates that the command is not pointing to the flow's file:
Copy code
docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.43/containers/44bd63ec74d08ad90a13770b140a149b3f7163491dcca3401242457cae2c7db4/start>: Internal Server Error ("failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "prefect": cannot run executable found relative to current directory: unknown")
✅ 1
a
Hey @Jon! 1. This will be released later today! 2. The command that it wants to run 
prefect flow-run execute
. Do you have 
prefect
installed in your image?
j
1 = awesome!
could we already achieve the same thing in the prefect.yaml implementation?
a
Yup you could by hardcoding 
image
in the 
job_variables
for a deployment and not including a 
build_docker_image
step.
✅ 1
j
2 - yup, prefect is installed on the image. perhaps it has something to do with the pathing?
Copy code
.
|- .venv
|- namespace/project/flow.py
a
Ah, is 
prefect
installed in a venv in your image?
j
yeah
docker run --rm -it --entrypoint bash _my-image_
when i hop into the image like that
and run 
prefect
, i get the CLI options
so i am confident that any prefect CLI command against the image should "just work"
or like... pretty confident 😅
a
What if you run 
docker run --rm my-image prefect --help
?
j
Copy code
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "prefect": cannot run executable found relative to current directory: unknown.
a
Huh, does your image have a default entrypoint?
j
nope
i use build stages, and in the final stage just copy over the dependencies
a
Maybe it would work with a default entrypoint of 
bash
. You could try 
docker run --rm --entrypoint bash my-image prefect --help
j
nothing else
Copy code
➜  data-capture git:(main) docker run --rm --entrypoint bash dc-pipelines prefect --help
./.venv/bin/prefect: line 3: import: command not found
./.venv/bin/prefect: line 4: import: command not found
./.venv/bin/prefect: line 5: from: command not found
./.venv/bin/prefect: prefect: line 7: syntax error near unexpected token `('
./.venv/bin/prefect: prefect: line 7: `    sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])'
a
Well that’s interesting! I’m assuming you have a reason to not use the base Prefect images, but might as well check to make sure.
j
hmph, we have a monorepo with multiple projects that have their own deployments. we share a Dockerfile template across each of them. lemme try this out with the prefect image in the final stage
so the final stage of our build looks like this:
Copy code
FROM python:3.11-slim AS runtime

WORKDIR /app

ARG PROJECT_NAME
ARG PROJECT_PATH

ENV PATH ./.venv/bin:$PATH

COPY --from=build-envconsul /usr/local/bin/envconsul /usr/local/bin/envconsul
COPY --from=build-project /app/projects/${PROJECT_NAME}/.venv/ ./.venv/
COPY ./data-capture/projects/${PROJECT_NAME}/dc/ ./dc/
when i use the prefect image
Copy code
FROM prefecthq/prefect:2-python3.11 AS runtime

WORKDIR /app

ARG PROJECT_NAME
ARG PROJECT_PATH

ENV PATH ./.venv/bin:$PATH

COPY --from=build-envconsul /usr/local/bin/envconsul /usr/local/bin/envconsul
COPY --from=build-project /app/projects/${PROJECT_NAME}/.venv/ ./.venv/
COPY ./data-capture/projects/${PROJECT_NAME}/dc/ ./dc/
the error is the same
Copy code
➜  data-capture git:(main) ✗ docker run --rm --entrypoint bash dc-pipelines prefect --help
./.venv/bin/prefect: line 3: import: command not found
./.venv/bin/prefect: line 4: import: command not found
./.venv/bin/prefect: line 5: from: command not found
./.venv/bin/prefect: prefect: line 7: syntax error near unexpected token `('
./.venv/bin/prefect: prefect: line 7: `    sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])'
a
What if you omit the 
bash
entrypoint? I think we need to find the right incantation to make sure the path update is respected.
j
sorry, i'm not following. what command are you thinking?
a
I’m thinking 
docker run --rm my-image prefect --help
when using the prefect base image.
j
Copy code
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "prefect": cannot run executable found relative to current directory: unknown.
a
Hmmm, I guess your venv in the path is blocking the base 
prefect
executable in the container. Can you try setting using an absolute path when setting 
PATH
? Like this 
ENV PATH /app/.venv/bin:$PATH
j
that did it
facepalm
let me try to run this guy again, thank you for your patience..
ok, it executed!
thank you
a
🎉 Glad it worked!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by