Ajeel Ahmed
05/18/2023, 10:08 PMDockerRegistry
block) through the image_registry
parameter of DockerContainer
doesn’t work, I can tell that by looking at the Docker registry field in the GUI is still empty. This means I have to manually go to the GUI and point it to the registry. I don’t want to set it myself each time, how do I accomplish this through the library?Nate
05/19/2023, 2:22 PMAjeel Ahmed
05/19/2023, 2:30 PMNate
05/19/2023, 2:33 PMwhen the docker registry field is empty it crashes so I have to go to GUI update the field myselfthis is surprising to me, because i see that the registry stays attached to the
DockerContainer
(below)
I would expect that when the agent pulls the DockerContainer
, that it still has the image_registry
filled out, even if the UI doesn't show it for some reason
In [5]: DockerContainer(image_registry=registry)
Out[5]: DockerContainer(type='docker-container', env={}, labels={}, name=None, command=None, image='prefecthq/prefect:2.10.8-python3.10', image_pull_policy=None, image_registry=DockerRegistry(username='zzstoatzz', password=SecretStr('**********'), registry_url='<http://registry.hub.docker.com|registry.hub.docker.com>', reauth=True), networks=[], network_mode=None, auto_remove=False, volumes=[], stream_output=True, memswap_limit=None, mem_limit=None, privileged=False)
In [6]: DockerContainer(image_registry=registry).save("test")
Out[6]: UUID('e49c230d-5017-4000-aae3-85346b55124f')
In [7]: exit
❯ ipython
In [1]: from prefect.infrastructure.docker import DockerContainer, DockerRegistry
In [2]: DockerContainer.load("test")
Out[2]: DockerContainer(type='docker-container', env={}, labels={}, name=None, command=None, image='prefecthq/prefect:2.10.8-python3.10', image_pull_policy=None, image_registry=DockerRegistry(username='zzstoatzz', password=SecretStr('**********'), registry_url='<http://registry.hub.docker.com|registry.hub.docker.com>', reauth=True), networks=[], network_mode=None, auto_remove=False, volumes=[], stream_output=True, memswap_limit=None, mem_limit=None, privileged=False)
Ajeel Ahmed
05/19/2023, 2:34 PMregistry_block = DockerRegistry.load("gitlab-docker-registry")
infra_block = DockerContainer(
image_registry=registry_block,
image="link to the image",
image_pull_policy="ALWAYS",
networks=[network stuff]
env=asdict(Config()),
auto_remove=True,
stream_output=True,
)
latest
? If so where am I supposed to put the link to the image?Nate
05/19/2023, 2:42 PMimage
wants something like registry/image:tag
can you share the logs from when it crashes?Ajeel Ahmed
05/19/2023, 2:42 PM2023-05-18 22:46:03 21:46:03.494 | DEBUG | prefect.agent - Checking for cancelled flow runs...
2023-05-18 22:46:13 21:46:13.368 | DEBUG | prefect.agent - Checking for cancelled flow runs...
2023-05-18 22:46:16 21:46:16.759 | DEBUG | prefect.agent - Checking for scheduled flow runs...
2023-05-18 22:46:27 21:46:27.000 | DEBUG | prefect.agent - Checking for scheduled flow runs...
2023-05-18 22:46:27 21:46:27.184 | DEBUG | prefect.agent - Checking for cancelled flow runs...
2023-05-18 22:46:35 21:46:35.382 | DEBUG | prefect.agent - Checking for scheduled flow runs...
2023-05-18 22:46:35 21:46:35.864 | INFO | prefect.agent - Submitting flow run '4e3de005-6c51-492d-8bf7-437c6088167c'
2023-05-18 22:46:36 21:46:36.954 | DEBUG | prefect.infrastructure.docker-container - Could not find Docker image locally: <http://registry.gitlab.com/planet-farms/periplo-innovation/project-precog/ml_db:latest|registry.gitlab.com/planet-farms/periplo-innovation/project-precog/ml_db:latest>
2023-05-18 22:46:36 21:46:36.958 | INFO | prefect.infrastructure.docker-container - Pulling image '<http://registry.gitlab.com/planet-farms/periplo-innovation/project-precog/ml_db:latest|registry.gitlab.com/planet-farms/periplo-innovation/project-precog/ml_db:latest>'...
2023-05-18 22:46:38 21:46:37.980 | ERROR | prefect.agent - Failed to submit flow run '4e3de005-6c51-492d-8bf7-437c6088167c' to infrastructure.
2023-05-18 22:46:38 Traceback (most recent call last):
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status
2023-05-18 22:46:38 response.raise_for_status()
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
2023-05-18 22:46:38 raise HTTPError(http_error_msg, response=self)
2023-05-18 22:46:38 requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=registry.gitlab.com%2Fplanet-farms%2Fperiplo-innovation%2Fproject-precog%2Fml_db>
2023-05-18 22:46:38
2023-05-18 22:46:38 The above exception was the direct cause of the following exception:
2023-05-18 22:46:38
2023-05-18 22:46:38 Traceback (most recent call last):
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 424, in _submit_run_and_capture_errors
2023-05-18 22:46:38 result = await infrastructure.run(task_status=task_status)
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 267, in run
2023-05-18 22:46:38 container = await run_sync_in_worker_thread(self._create_and_start_container)
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
2023-05-18 22:46:38 return await anyio.to_thread.run_sync(
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
2023-05-18 22:46:38 return await get_asynclib().run_sync_in_worker_thread(
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
2023-05-18 22:46:38 return await future
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
2023-05-18 22:46:38 result = context.run(func, *args)
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 365, in _create_and_start_container
2023-05-18 22:46:38 self._pull_image(docker_client)
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 475, in _pull_image
2023-05-18 22:46:38 return docker_client.images.pull(image, tag)
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull
2023-05-18 22:46:38 pull_log = self.client.api.pull(
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/docker/api/image.py", line 429, in pull
2023-05-18 22:46:38 self._raise_for_status(response)
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status
2023-05-18 22:46:38 raise create_api_error_from_http_exception(e) from e
2023-05-18 22:46:38 File "/usr/local/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
2023-05-18 22:46:38 raise cls(e, response=response, explanation=explanation) from e
2023-05-18 22:46:38 docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=registry.gitlab.com%2Fplanet-farms%2Fperiplo-innovation%2Fproject-precog%2Fml_db>: Internal Server Error ("Head "<https://registry.gitlab.com/v2/planet-farms/periplo-innovation/project-precog/ml_db/manifests/latest>": denied: access forbidden")
Nate
05/19/2023, 2:47 PMAjeel Ahmed
05/19/2023, 2:55 PMNate
05/19/2023, 2:56 PMAjeel Ahmed
05/19/2023, 2:57 PMNate
05/19/2023, 2:57 PMAjeel Ahmed
05/22/2023, 10:57 AMNate
05/22/2023, 1:49 PM