Ajeel Ahmed
05/17/2023, 10:53 AMprefect.infrastructure
’s DockerContainer
and I want to set its image_pull_policy
to ALWAYS
but for some reason it fails to pull the image from the registryTaylor Curran
05/17/2023, 12:59 PMAjeel Ahmed
05/17/2023, 1:00 PM13:11:53.658 | INFO | prefect.agent - Submitting flow run '7111f659-a6e5-4a17-aef6-d06faecf9b5d'
13:11:53.783 | INFO | prefect.infrastructure.docker-container - Pulling image '<http://registry.gitlab.com/planet-farms/project/ml_db:latest|registry.gitlab.com/planet-farms/project/ml_db:latest>'...
13:11:54.464 | ERROR | prefect.agent - Failed to submit flow run '7111f659-a6e5-4a17-aef6-d06faecf9b5d' to infrastructure.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=registry.gitlab.com%2Finnovation%2Fproject-%2Fml_db>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 424, in _submit_run_and_capture_errors
result = await infrastructure.run(task_status=task_status)
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 267, in run
container = await run_sync_in_worker_thread(self._create_and_start_container)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 365, in _create_and_start_container
self._pull_image(docker_client)
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 475, in _pull_image
return docker_client.images.pull(image, tag)
File "/usr/local/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull
pull_log = self.client.api.pull(
File "/usr/local/lib/python3.10/site-packages/docker/api/image.py", line 429, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e) from e
File "/usr/local/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=registry.gitlab.com%2Fps%-innovation%2Fproject-precog%2Fml_db>: Internal Server Error ("Head <https://registry.gitlab.com/v2/-farms//ml_db/manifests/latest>: denied: access forbidden")
13:11:54.480 | INFO | prefect.agent - Completed submission of flow run '7111f659-a6e5-4a17-aef6-d06faecf9b5d'
denied: access forbidden
500 errorTaylor Curran
05/17/2023, 2:35 PMAjeel Ahmed
05/17/2023, 2:35 PMNate
05/17/2023, 2:38 PM"Head <https://registry.gitlab.com/v2/-farms//ml_db/manifests/latest>: denied: access forbidden"
does this url look right to you?Ajeel Ahmed
05/17/2023, 2:40 PMNate
05/17/2023, 2:40 PM.login()
on your DockerRegistry
block and return an authed Docker client? I agree with taylor that it looks like some sort of auth issueAjeel Ahmed
05/17/2023, 2:47 PMNate
05/17/2023, 2:57 PMAjeel Ahmed
05/17/2023, 3:04 PMNate
05/17/2023, 3:06 PMAjeel Ahmed
05/17/2023, 3:07 PMNate
05/17/2023, 3:08 PMAjeel Ahmed
05/17/2023, 3:16 PMprefect.agent - Reported flow run 'e4a1edf5-0069-4498-9b86-a0cbb7ec4532' as crashed: Flow run infrastructure exited with non-zero status code 1
Nate
05/17/2023, 3:22 PMAjeel Ahmed
05/17/2023, 3:23 PMNate
05/17/2023, 3:24 PM