Hello there I try to use new concept of projects and to deploy with

prefect_docker

template to build a docker image and to push it on my private registry (on Gcloud) but it looks like it's not possible to use an other registry than docker. Any idea? Otherwise, what's the best practice to deploy a docker image flow on a private storage?

Hey @Florent VanDeMoortele! If you authenticate with your private Google Cloud registry via the Docker CLI before running deploy for your project you should be able to push your image to your private registry. We use the

docker_build_image

this way internally.I expect we’ll add native support for more registry types in the future, but authenticating via the Docker CLI is the best way to do it right now.

Thank you for your reply @alex. I try it but I can see where I specify my registry URL. I obtain this error:

docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.42/auth>: Internal Server Error ("Get "<https://eu.gcr.io/v2/>": unauthorized: Not Authorized.")

Can you share the commands that you used to login with the Docker CLI?

For sure, here is my

prefect.yaml

# build section allows you to manage and build docker images
build:
    - prefect_docker.projects.steps.build_docker_image:
        requires: prefect-docker
        image_name: image_test
        tag: dev
        dockerfile: Dockerfile
        push: True
        credentials: "{{ prefect.blocks.docker-registry-credentials.dev-registry }}"

I think you’ll need to remove your

credentials

so that the Docker client will use the credentials from your machine. Could you try again with your

credentials

removed from your

prefect.yaml

?

thank you, it works ! My docker image is now uploaded on my storage. But if I remove my

credentials

, how the flow can pull the image? When I launch this flow on the cloud UI, I have this error:

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 276, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/client/utilities.py", line 40, in with_injected_client
    return await fn(*args, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/deployments.py", line 205, in load_flow_from_flow_run
    output.update(await run_step(step))
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/projects/steps/core.py", line 78, in run_step
    return await from_async.call_soon_in_new_thread(
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 181, in aresult
    return await asyncio.wrap_future(self.future)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 194, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/projects/steps/pull.py", line 26, in set_working_directory
    os.chdir(directory)
FileNotFoundError: [Errno 2] No such file or directory

The permissions for pulling images are handled by the worker or service running the flow. What work pool type are you using for this deployment?

Actually I use a docker worker

OK, you can set the image job variable to tell your Docker worker to use your custom-built image. In your

deployment.yaml

file, it’d look like this:

work_pool:
  name: your-work-pool-name
  job_variables:
    image: {{ image_name }}

That will take the name of the created image output by the

build_docker_image

step and set it on your deployment so that your Docker worker knows which image to pull.

Yes, but I don't understand how the worker can access to my private registry. Because when I deploy I have the credentials on my machine, and with your hack I can push on the private registry. But when I launch the flow, the flow doesn't have these credentials, so I have this error:

docker.errors.ImageNotFound: 404 Client Error for <http+docker://localhost/v1.42/images/create?tag=e1c999a55b4e0eb28cc9db90a26c883bebd472423c89d25469bd0960c367c175&fromImage=sha256>: Not Found ("pull access denied for sha256, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

What type of work pool are you using? Depending on the work pool type, you can provide registry credentials on the work pool, which the work will use to pull images. The error that you’re seeing makes me think that you don’t have a pull step in either your

prefect.yaml

or

deployment.yaml

file. Do you have a pull step defined?

My bad !

prefect.projects.steps.set_working_directory

was missing, thank you @alex !! Now it works when I launch locally my worker. I'll try to deploy a worker on my GCP infrastructure and to launch this flow. But I'm not sure of what I'm doing with new concepts on V2. On Prefect V1, I use to deploy an agent for each project and every flows of this project run with this agent. Have you an advice for Prefect V2?