William Grim
02/24/2022, 8:45 PMDockerStorage
and then realized I can't use a local
agent to run flows that use that. No problem... since my local agent is setup in a docker container, I figured I would just use a DockerRun
config. This seems to go alright at the very first, but then it can't pull the image from our ECR. đź§µWilliam Grim
02/24/2022, 8:46 PMDuring handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 391, in _deploy_flow_run
deployment_info = self.deploy_flow(flow_run)
File "/usr/local/lib/python3.8/site-packages/prefect/agent/docker/agent.py", line 384, in deploy_flow
pull_output = self.docker_client.pull(image, stream=True, decode=True)
File "/usr/local/lib/python3.8/site-packages/docker/api/image.py", line 430, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=my_account_id.dkr.ecr.us-wes>
t-1.amazonaws.com%2Fpath%2Fto%2Fimage: Internal Server Error ("Head "<https://my_account_id.dkr.ecr.us-west-1.amazo>
<http://naws.com/v2/path/to/image/manifests/latest|naws.com/v2/path/to/image/manifests/latest>": no basic auth credentials")
William Grim
02/24/2022, 8:47 PMAWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
exist in the environment the agent is running in and in the location where I did flow registration from. Is there something else to do to get DockerRun
or docker agent to pull images from our private ECR?Anna Geller
Anna Geller
aws ecr get-login-password --region region | docker login --username AWS --password-stdin <http://aws_account_id.dkr.ecr.region.amazonaws.com|aws_account_id.dkr.ecr.region.amazonaws.com>
Anna Geller
William Grim
02/24/2022, 8:54 PMlocal
agent, but i thought that since it's not compatible with DockerStorage
, i need to switch to DockerRun
and docker agent, especially since our prod is k8s and we want to get prefect submitting k8s jobs instead of running local processes.Anna Geller
William Grim
02/24/2022, 8:55 PMWilliam Grim
02/24/2022, 8:56 PMWilliam Grim
02/24/2022, 8:56 PMAnna Geller
William Grim
02/24/2022, 8:57 PMAnna Geller
William Grim
02/24/2022, 8:58 PMKubernetesAgent
if we want to have prefect inside the container. i saw that KubernetesRun
can have image_pull_secrets
set. i just wasn't sure if that kind of agent was going to work in dockerWilliam Grim
02/24/2022, 8:58 PMAnna Geller
KubernetesAgent
for your dev environment as well - you can pull secrets this way, that’s correct.William Grim
02/24/2022, 8:59 PMWilliam Grim
02/24/2022, 8:59 PMAnna Geller
William Grim
02/24/2022, 9:00 PMAnna Geller
William Grim
02/25/2022, 10:26 PM