https://prefect.io logo
a

Ajeel Ahmed

05/18/2023, 10:08 PM
Hello again! Yes I’m back 😅 The problem: Pointing to the private registry (a
DockerRegistry
block) through the
image_registry
parameter of
DockerContainer
doesn’t work, I can tell that by looking at the Docker registry field in the GUI is still empty. This means I have to manually go to the GUI and point it to the registry. I don’t want to set it myself each time, how do I accomplish this through the library?
1
n

Nate

05/19/2023, 2:22 PM
hi @Ajeel Ahmed - can you share what you're trying programmatically that you're saying has no effect?
hmm okay I've reproduced this, i think this is a UI problem - when I load the saved block programmatically, it still has the registry attached
a

Ajeel Ahmed

05/19/2023, 2:30 PM
it does not update yes, so when I try to run the flow when the docker registry field is empty it crashes so I have to go to GUI update the field myself and then run the flow and then it does work
I hope you get what I’m trying to say haha
n

Nate

05/19/2023, 2:33 PM
when the docker registry field is empty it crashes so I have to go to GUI update the field myself
this is surprising to me, because i see that the registry stays attached to the
DockerContainer
(below) I would expect that when the agent pulls the
DockerContainer
, that it still has the
image_registry
filled out, even if the UI doesn't show it for some reason
Copy code
In [5]: DockerContainer(image_registry=registry)
Out[5]: DockerContainer(type='docker-container', env={}, labels={}, name=None, command=None, image='prefecthq/prefect:2.10.8-python3.10', image_pull_policy=None, image_registry=DockerRegistry(username='zzstoatzz', password=SecretStr('**********'), registry_url='<http://registry.hub.docker.com|registry.hub.docker.com>', reauth=True), networks=[], network_mode=None, auto_remove=False, volumes=[], stream_output=True, memswap_limit=None, mem_limit=None, privileged=False)

In [6]: DockerContainer(image_registry=registry).save("test")
Out[6]: UUID('e49c230d-5017-4000-aae3-85346b55124f')

In [7]: exit

❯ ipython

In [1]: from prefect.infrastructure.docker  import DockerContainer, DockerRegistry

In [2]: DockerContainer.load("test")
Out[2]: DockerContainer(type='docker-container', env={}, labels={}, name=None, command=None, image='prefecthq/prefect:2.10.8-python3.10', image_pull_policy=None, image_registry=DockerRegistry(username='zzstoatzz', password=SecretStr('**********'), registry_url='<http://registry.hub.docker.com|registry.hub.docker.com>', reauth=True), networks=[], network_mode=None, auto_remove=False, volumes=[], stream_output=True, memswap_limit=None, mem_limit=None, privileged=False)
a

Ajeel Ahmed

05/19/2023, 2:34 PM
Copy code
registry_block = DockerRegistry.load("gitlab-docker-registry")

    infra_block = DockerContainer(
        image_registry=registry_block,
        image="link to the image",
        image_pull_policy="ALWAYS",
        networks=[network stuff]
        env=asdict(Config()),
        auto_remove=True,
        stream_output=True,
    )
this is the code
here’s the thing: the documentation says the image parameter should take in the tag of the docker image, does that mean I’m supposed to only put the tag name? Like
latest
? If so where am I supposed to put the link to the image?
you can see it here
n

Nate

05/19/2023, 2:42 PM
im pretty sure that docstring is slightly misleading and
image
wants something like
registry/image:tag
can you share the logs from when it crashes?
a

Ajeel Ahmed

05/19/2023, 2:42 PM
yeah sure just give me a sec
Copy code
2023-05-18 22:46:03 21:46:03.494 | DEBUG   | prefect.agent - Checking for cancelled flow runs...
2023-05-18 22:46:13 21:46:13.368 | DEBUG   | prefect.agent - Checking for cancelled flow runs...
2023-05-18 22:46:16 21:46:16.759 | DEBUG   | prefect.agent - Checking for scheduled flow runs...
2023-05-18 22:46:27 21:46:27.000 | DEBUG   | prefect.agent - Checking for scheduled flow runs...
2023-05-18 22:46:27 21:46:27.184 | DEBUG   | prefect.agent - Checking for cancelled flow runs...
2023-05-18 22:46:35 21:46:35.382 | DEBUG   | prefect.agent - Checking for scheduled flow runs...
2023-05-18 22:46:35 21:46:35.864 | INFO    | prefect.agent - Submitting flow run '4e3de005-6c51-492d-8bf7-437c6088167c'
2023-05-18 22:46:36 21:46:36.954 | DEBUG   | prefect.infrastructure.docker-container - Could not find Docker image locally: <http://registry.gitlab.com/planet-farms/periplo-innovation/project-precog/ml_db:latest|registry.gitlab.com/planet-farms/periplo-innovation/project-precog/ml_db:latest>
2023-05-18 22:46:36 21:46:36.958 | INFO    | prefect.infrastructure.docker-container - Pulling image '<http://registry.gitlab.com/planet-farms/periplo-innovation/project-precog/ml_db:latest|registry.gitlab.com/planet-farms/periplo-innovation/project-precog/ml_db:latest>'...
2023-05-18 22:46:38 21:46:37.980 | ERROR   | prefect.agent - Failed to submit flow run '4e3de005-6c51-492d-8bf7-437c6088167c' to infrastructure.
2023-05-18 22:46:38 Traceback (most recent call last):
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status
2023-05-18 22:46:38     response.raise_for_status()
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
2023-05-18 22:46:38     raise HTTPError(http_error_msg, response=self)
2023-05-18 22:46:38 requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=registry.gitlab.com%2Fplanet-farms%2Fperiplo-innovation%2Fproject-precog%2Fml_db>
2023-05-18 22:46:38 
2023-05-18 22:46:38 The above exception was the direct cause of the following exception:
2023-05-18 22:46:38 
2023-05-18 22:46:38 Traceback (most recent call last):
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 424, in _submit_run_and_capture_errors
2023-05-18 22:46:38     result = await infrastructure.run(task_status=task_status)
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 267, in run
2023-05-18 22:46:38     container = await run_sync_in_worker_thread(self._create_and_start_container)
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
2023-05-18 22:46:38     return await anyio.to_thread.run_sync(
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
2023-05-18 22:46:38     return await get_asynclib().run_sync_in_worker_thread(
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
2023-05-18 22:46:38     return await future
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
2023-05-18 22:46:38     result = context.run(func, *args)
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 365, in _create_and_start_container
2023-05-18 22:46:38     self._pull_image(docker_client)
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 475, in _pull_image
2023-05-18 22:46:38     return docker_client.images.pull(image, tag)
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull
2023-05-18 22:46:38     pull_log = self.client.api.pull(
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/docker/api/image.py", line 429, in pull
2023-05-18 22:46:38     self._raise_for_status(response)
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status
2023-05-18 22:46:38     raise create_api_error_from_http_exception(e) from e
2023-05-18 22:46:38   File "/usr/local/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
2023-05-18 22:46:38     raise cls(e, response=response, explanation=explanation) from e
2023-05-18 22:46:38 docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=registry.gitlab.com%2Fplanet-farms%2Fperiplo-innovation%2Fproject-precog%2Fml_db>: Internal Server Error ("Head "<https://registry.gitlab.com/v2/planet-farms/periplo-innovation/project-precog/ml_db/manifests/latest>": denied: access forbidden")
this is the error when the field for the Docker Registry is empty
so when I fill it with the Docker Registry block it’s supposed to have (the one that is in the code above) and run the flow again it runs just fine
n

Nate

05/19/2023, 2:47 PM
hmm this seems like a bug then. would you be willing to open an issue?
a

Ajeel Ahmed

05/19/2023, 2:55 PM
oh yeah of course! I was talking to a coworker and I was sure we were doing something wrong but good to know this is probably a bug!
n

Nate

05/19/2023, 2:56 PM
i could be missing something as well, but its at least a little weird, since the registry block should definitely appear in the UI after saving the block with the registry attached
a

Ajeel Ahmed

05/19/2023, 2:57 PM
yeah we can close it if it’s something on my side
n

Nate

05/19/2023, 2:57 PM
👍
🙌 1
a

Ajeel Ahmed

05/22/2023, 10:57 AM
Hey @Nate hope you had a great weekend, I thought I’d investigate this issue a bit more before making an issue on GitHub and fortunately we realised the mistake, our setup for blocks was failing and we weren’t getting notified so it was not updating the blocks that were already there, which made me think there was something wrong. Fixed now, Prefect works just fine! Thanks again.
🙌 1
n

Nate

05/22/2023, 1:49 PM
thanks for digging into that and letting me know 🙌 appreciate the update - glad things are working!
🙌 1