https://prefect.io logo
Title
w

William Grim

02/24/2022, 8:45 PM
Alright, I'm slowly pushing further with
DockerStorage
and then realized I can't use a
local
agent to run flows that use that. No problem... since my local agent is setup in a docker container, I figured I would just use a
DockerRun
config. This seems to go alright at the very first, but then it can't pull the image from our ECR. 🧵
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 391, in _deploy_flow_run
    deployment_info = self.deploy_flow(flow_run)
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/docker/agent.py", line 384, in deploy_flow
    pull_output = self.docker_client.pull(image, stream=True, decode=True)
  File "/usr/local/lib/python3.8/site-packages/docker/api/image.py", line 430, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=my_account_id.dkr.ecr.us-wes>
t-1.amazonaws.com%2Fpath%2Fto%2Fimage: Internal Server Error ("Head "<https://my_account_id.dkr.ecr.us-west-1.amazo>
<http://naws.com/v2/path/to/image/manifests/latest|naws.com/v2/path/to/image/manifests/latest>": no basic auth credentials")
The
AWS_ACCESS_KEY_ID
and
AWS_SECRET_ACCESS_KEY
exist in the environment the agent is running in and in the location where I did flow registration from. Is there something else to do to get
DockerRun
or docker agent to pull images from our private ECR?
a

Anna Geller

02/24/2022, 8:52 PM
So in the end you use a docker agent and Docker storage?
to authenticate your Docker agent with ECR, you can run the command:
aws ecr get-login-password --region region | docker login --username AWS --password-stdin <http://aws_account_id.dkr.ecr.region.amazonaws.com|aws_account_id.dkr.ecr.region.amazonaws.com>
you would need to run the above command in your terminal before you start your Docker agent. This will update your Docker config file with the ECR credentials and will ensure that flow runs spun up as docker containers by your Docker agent will (should) be able to pull the image
w

William Grim

02/24/2022, 8:54 PM
@Anna Geller well, we bring up a stack of docker containers, and one of those containers is where we run a prefect agent. it was running as a
local
agent, but i thought that since it's not compatible with
DockerStorage
, i need to switch to
DockerRun
and docker agent, especially since our prod is k8s and we want to get prefect submitting k8s jobs instead of running local processes.
a

Anna Geller

02/24/2022, 8:55 PM
Note this caveat here - the docker agent must run as a local process, not as a Docker container, otherwise you’ll get into trouble when scaling your flows. The same with a local agent actually, it should run as a local process and you can run it with supervisor to make it more “robust” and allowing the agent to restart itself
w

William Grim

02/24/2022, 8:55 PM
oh yeah, we bring up like a more full-featured container, and inside of that container, we start a process that is the prefect agent
is that what you mean?
oh, i'll read what you posted
a

Anna Geller

02/24/2022, 8:56 PM
but you then run docker in docker, I’d encourage you to check up the thread I shared and to run the agent as a local process to avoid issues
w

William Grim

02/24/2022, 8:57 PM
a local process where? on the host machine?
a

Anna Geller

02/24/2022, 8:57 PM
exactly, on some VM where you would like to deploy it. Alternative: you spin up a separate KubernetesAgent e.g. in “dev” namespace for your non-prod environment - may actually get easier
w

William Grim

02/24/2022, 8:58 PM
i'm reading on that link that we should use
KubernetesAgent
if we want to have prefect inside the container. i saw that
KubernetesRun
can have
image_pull_secrets
set. i just wasn't sure if that kind of agent was going to work in docker
yeah, moving prefect outside docker and onto host machines is too big a dev change if we can use the k8s agent. think that's possible in docker?
a

Anna Geller

02/24/2022, 8:59 PM
yes, you’re spot on, in your use case it makes sense to use a
KubernetesAgent
for your dev environment as well - you can pull secrets this way, that’s correct.
w

William Grim

02/24/2022, 8:59 PM
freakin' awesome
since we want to get prod to that end result, that's nice i can try it in dev
a

Anna Geller

02/24/2022, 8:59 PM
if you need an example for the Secret setup, check this Readme and flow example in this repo https://github.com/anna-geller/packaging-prefect-flows/
w

William Grim

02/24/2022, 9:00 PM
Thanks @Anna Geller; all your help the last couple days has been great. I'm gonna go through this and hopefully push us over the line.
a

Anna Geller

02/24/2022, 9:01 PM
sure, keep us posted if you have any questions along the way
👍 1
w

William Grim

02/25/2022, 10:26 PM
@Christian Nuss