I am trying to replicate running this flow on my own laptop with Docker Desktop. I have created a docker image (but did not push it to e.g. DockerHub) and registered the flow just like in the script to Prefect Cloud. Started a docker agent and did a quick run from the UI. So far so good everything works. But, when I modify the Dockerfile so that it can run as non-root, I can no longer do a successful run. I get the following error from the agent:

docker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/f30a3e9241fab0d272c3f36eae16867487fb187b964c8b3f22bc8fd05d2aa4d0/json>: Not Found ("No such container: f30a3e9241fab0d272c3f36eae16867487fb187b964c8b3f22bc8fd05d2aa4d0")

and the flow is stuck in submitted state. Does anyone knows how to fix it? Just to make sure, the only difference between the successful run and the run stuck in submitted state is the fact that I added a non-root user to the image. https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows_no_build/docker_script_docker_run_local_image.py

Nice work! It makes sense because Prefect sets the ENTRYPOINT itself to trigger the flow run. I never set any entrypoints when building images for Prefect flows

That's very strange that Prefect does not overwrite the image's default entrypoint then, no?

I think it's the other way around 🙂 you use Prefect's base image and you actually have overwritten the entrypoint set by Prefect, that's why it didn't work for you. If you don't set it, you keep the entrypoint set in the Prefect's base image

That's true in this example, but the use case I have in mind is that I want to start from a regular Python base image. The reason for this is that some companies only offer a curated list of base images that you are allowed to use. Hence you cannot use Prefect’s base image and have to start from scratch. In that case, you want to make sure the entrypoint is properly set. In the job spec of the k8s runconfig, this is indeed the case https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/kubernetes/job_spec.yaml#L15

I think to use your own custom base image, it would be easier to replicate Prefect's Dockerfile using your custom base image instead of changing commands used by Docker/Kubernetes agent, right? Also, the entrypoint is doing a bit more, e.g. allows to install packages set on env variable:

pip install $EXTRA_PIP_PACKAGES

Lastly, Prefect is using the base Python image in the slim version, which is pretty standard and commonly accepted for Python applications - what else would you want to use instead (e.g. Ubuntu base-image and install Python yourself)? Having said that, feel free to submit a PR or GitHub issue and we can discuss it further with other engineers, if you feel like this is a problem we should address.

I’m guessing that indeed replicating Prefect’s Dockerfile solves the issue! Thanks for the advice!