m

    Mac Gréco Péralte Chéry

    1 year ago
    Hello Eveveryone! I am using a CI/CD process with AWS Codepipeline (source: Github, build: Codebuild) the output is a docker image i push to an AWS ECR repository for each flow. I my github project i have a buildspec file where i put this code:
    build:
        commands:
          - python --version
          - pyenv global 3.7.7
          - python --version
          - pip install $(grep -ivE "pywin32" requirements.txt)
          - prefect backend cloud
          - prefect auth login -t $PREFECT_ACCESS_TOKEN
          - aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin <http://xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com|xxxxxxxx.dkr.ecr.us-east-2.amazonaws.com>
          - python3.7 flows/my_flow.py
    From November 20 docker introduced pull limits for images what cause me to have this error in Code build
    File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 619, in pull_image
        output = client.pull(self.base_image, stream=True, decode=True)
      File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/api/image.py", line 415, in pull
        self._raise_for_status(response)
      File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/api/client.py", line 261, in _raise_for_status
        raise create_api_error_from_http_exception(e)
      File "/root/.pyenv/versions/3.7.7/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
        raise cls(e, response=response, explanation=explanation)
    docker.errors.APIError: 500 Server Error: Internal Server Error ("toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: <https://www.docker.com/increase-rate-limit>")
    When i go to the site i can read: "Anonymous free users will be limited to 100 pulls per six hours, and authenticated free users will be limited to 200 pulls per six hours." So i created a account on docker to raise my limits to 200 pull per 6 hours. So i understand the problem occurs when prefect is pulling the base prefect docker image to build my flow image because i have reached my 100 pulls / 6 hours anonymous limit My question is where should i put my docker account token to pull the prefect base image via my newly created docker account?
    Michael Adkins

    Michael Adkins

    1 year ago
    Hi! We’re just using the
    docker-py
    API package. It provides a
    login
    method, but we haven’t set anything up to use that. I’m curious if you use the
    docker login
    command before building your flow if it’ll store and use the credentials.
    Alternatively, you could consider copying the base image into ECR and change the base location in the
    Docker
    storage class to reference that and avoid rate limiting entirely?
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    To clarify Michael's 2nd suggestion, you can copy the
    prefecthq/prefect
    base image into ECR, then pass the registry & name for the ECR version as the
    base_image
    kwarg to the
    Docker
    storage. This would let you avoid ever hitting dockerhub.
    m

    Mac Gréco Péralte Chéry

    1 year ago
    Wonderful!. I just tested the 2nd suggestion. It works flawlessly. Thanks a lot @Michael Adkins and @Jim Crist-Harif.