William Grim
02/24/2022, 8:45 PMDockerStorage
and then realized I can't use a local
agent to run flows that use that. No problem... since my local agent is setup in a docker container, I figured I would just use a DockerRun
config. This seems to go alright at the very first, but then it can't pull the image from our ECR. đź§µDuring handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 391, in _deploy_flow_run
deployment_info = self.deploy_flow(flow_run)
File "/usr/local/lib/python3.8/site-packages/prefect/agent/docker/agent.py", line 384, in deploy_flow
pull_output = self.docker_client.pull(image, stream=True, decode=True)
File "/usr/local/lib/python3.8/site-packages/docker/api/image.py", line 430, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=my_account_id.dkr.ecr.us-wes>
t-1.amazonaws.com%2Fpath%2Fto%2Fimage: Internal Server Error ("Head "<https://my_account_id.dkr.ecr.us-west-1.amazo>
<http://naws.com/v2/path/to/image/manifests/latest|naws.com/v2/path/to/image/manifests/latest>": no basic auth credentials")
AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
exist in the environment the agent is running in and in the location where I did flow registration from. Is there something else to do to get DockerRun
or docker agent to pull images from our private ECR?Anna Geller
02/24/2022, 8:52 PMaws ecr get-login-password --region region | docker login --username AWS --password-stdin <http://aws_account_id.dkr.ecr.region.amazonaws.com|aws_account_id.dkr.ecr.region.amazonaws.com>
William Grim
02/24/2022, 8:54 PMlocal
agent, but i thought that since it's not compatible with DockerStorage
, i need to switch to DockerRun
and docker agent, especially since our prod is k8s and we want to get prefect submitting k8s jobs instead of running local processes.Anna Geller
02/24/2022, 8:55 PMWilliam Grim
02/24/2022, 8:55 PMAnna Geller
02/24/2022, 8:56 PMWilliam Grim
02/24/2022, 8:57 PMAnna Geller
02/24/2022, 8:57 PMWilliam Grim
02/24/2022, 8:58 PMKubernetesAgent
if we want to have prefect inside the container. i saw that KubernetesRun
can have image_pull_secrets
set. i just wasn't sure if that kind of agent was going to work in dockerAnna Geller
02/24/2022, 8:59 PMKubernetesAgent
for your dev environment as well - you can pull secrets this way, that’s correct.William Grim
02/24/2022, 8:59 PMAnna Geller
02/24/2022, 8:59 PMWilliam Grim
02/24/2022, 9:00 PMAnna Geller
02/24/2022, 9:01 PMWilliam Grim
02/25/2022, 10:26 PM