https://prefect.io logo
Powered by
Title
a

Ajeel Ahmed

05/17/2023, 10:53 AM
so I’m using 
prefect.infrastructure
’s 
DockerContainer
and I want to set its 
image_pull_policy
to 
ALWAYS
but for some reason it fails to pull the image from the registry
t

Taylor Curran

05/17/2023, 12:59 PM
Yes Ajeel, let us know what the error is — its often a permissions thing — like the agent or worker doesn’t have permissions to pull from your registry.
a

Ajeel Ahmed

05/17/2023, 1:00 PM
hey thanks for the reply! I’m replicating the error right now and will post it as soon as I get it.
hey @Taylor Curran here is the stack trace
13:11:53.658 | INFO    | prefect.agent - Submitting flow run '7111f659-a6e5-4a17-aef6-d06faecf9b5d'
13:11:53.783 | INFO    | prefect.infrastructure.docker-container - Pulling image '<http://registry.gitlab.com/planet-farms/project/ml_db:latest|registry.gitlab.com/planet-farms/project/ml_db:latest>'...
13:11:54.464 | ERROR   | prefect.agent - Failed to submit flow run '7111f659-a6e5-4a17-aef6-d06faecf9b5d' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=registry.gitlab.com%2Finnovation%2Fproject-%2Fml_db>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 424, in _submit_run_and_capture_errors
    result = await infrastructure.run(task_status=task_status)
  File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 267, in run
    container = await run_sync_in_worker_thread(self._create_and_start_container)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 365, in _create_and_start_container
    self._pull_image(docker_client)
  File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 475, in _pull_image
    return docker_client.images.pull(image, tag)
  File "/usr/local/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull
    pull_log = self.client.api.pull(
  File "/usr/local/lib/python3.10/site-packages/docker/api/image.py", line 429, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
  File "/usr/local/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=registry.gitlab.com%2Fps%-innovation%2Fproject-precog%2Fml_db>: Internal Server Error ("Head <https://registry.gitlab.com/v2/-farms//ml_db/manifests/latest>: denied: access forbidden")
13:11:54.480 | INFO    | prefect.agent - Completed submission of flow run '7111f659-a6e5-4a17-aef6-d06faecf9b5d'
the bottom line is 
denied: access forbidden
500 error
t

Taylor Curran

05/17/2023, 2:35 PM
What is the registry you are using? Looks like you have not authenticated to it.
a

Ajeel Ahmed

05/17/2023, 2:35 PM
that’s the thing I am authenticated afaik
I’m using gitlab registry
and I’m creating docker registry block for it with my username and password
like so
n

Nate

05/17/2023, 2:38 PM
"Head <https://registry.gitlab.com/v2/-farms//ml_db/manifests/latest>: denied: access forbidden"
does this url look right to you?
a

Ajeel Ahmed

05/17/2023, 2:40 PM
well I kinda removed some parts for privacy reasons but yes, it is for me
n

Nate

05/17/2023, 2:40 PM
gotcha
are you able to call 
.login()
on your 
DockerRegistry
block and return an authed Docker client? I agree with taylor that it looks like some sort of auth issue
a

Ajeel Ahmed

05/17/2023, 2:47 PM
let me try it out and report back, thanks guys
n

Nate

05/17/2023, 2:57 PM
👍
a

Ajeel Ahmed

05/17/2023, 3:04 PM
hey is the docker registry supposed to be empty like this?
n

Nate

05/17/2023, 3:06 PM
i assume this is your DockerContainer block? I would say no, if you're trying to use the DockerRegistry block to auth/pull your DockerContainer
a

Ajeel Ahmed

05/17/2023, 3:07 PM
yes it is my DockerContainer block
n

Nate

05/17/2023, 3:08 PM
gotcha, you should be able to edit your DockerContainer block and select your DockerRegistry so it can be used for auth
a

Ajeel Ahmed

05/17/2023, 3:16 PM
awesome! I’m getting a different error now
prefect.agent - Reported flow run 'e4a1edf5-0069-4498-9b86-a0cbb7ec4532' as crashed: Flow run infrastructure exited with non-zero status code 1
ah nevermind this was one is on my side
thank you so much @Nate and @Taylor Curran! You guys were great help!
n

Nate

05/17/2023, 3:22 PM
🎉
you're welcome! let us know if you run into anything else 🙂
a

Ajeel Ahmed

05/17/2023, 3:23 PM
I’ll try not to bother you guys too much 😆
n

Nate

05/17/2023, 3:24 PM
haha that's what this slack community is for! feel free :marvin:
🙏 1
#prefect-communityJoin Slack
Powered by