Hello Eveveryone! I am using a CI/CD process with...
# prefect-community
Hello Eveveryone! I am using a CI/CD process with AWS Codepipeline (source: Github, build: Codebuild) the output is a docker image i push to an AWS ECR repository for each flow. I my github project i have a buildspec file where i put this code:
Copy code
      - python --version
      - pyenv global 3.7.7
      - python --version
      - pip install $(grep -ivE "pywin32" requirements.txt)
      - prefect backend cloud
      - prefect auth login -t $PREFECT_ACCESS_TOKEN
      - aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin <http://xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com|xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com>
      - python3.7 flows/my_flow.py
From November 20 docker introduced pull limits for images what cause me to have this error in Code build
Copy code
File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 619, in pull_image
    output = client.pull(self.base_image, stream=True, decode=True)
  File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/api/image.py", line 415, in pull
  File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/api/client.py", line 261, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: <https://www.docker.com/increase-rate-limit>")
When i go to the site i can read: "Anonymous free users will be limited to 100 pulls per six hours, and authenticated free users will be limited to 200 pulls per six hours." So i created a account on docker to raise my limits to 200 pull per 6 hours. So i understand the problem occurs when prefect is pulling the base prefect docker image to build my flow image because i have reached my 100 pulls / 6 hours anonymous limit My question is where should i put my docker account token to pull the prefect base image via my newly created docker account?
Hi! We’re just using the
API package. It provides a
method, but we haven’t set anything up to use that. I’m curious if you use the
docker login
command before building your flow if it’ll store and use the credentials.
Alternatively, you could consider copying the base image into ECR and change the base location in the
storage class to reference that and avoid rate limiting entirely?
To clarify Michael's 2nd suggestion, you can copy the
base image into ECR, then pass the registry & name for the ECR version as the
kwarg to the
storage. This would let you avoid ever hitting dockerhub.
upvote 1
Wonderful!. I just tested the 2nd suggestion. It works flawlessly. Thanks a lot @Zanie and @Jim Crist-Harif.