Alright I m slowly pushing further with `DockerStorage` and

Question

Alright I m slowly pushing further with `DockerStorage` and then realized I can t use a `local` agent to run flows that use that No problem since my local agent is setup in a docker container I figured I would just use a `DockerRun` config This seems to go alright at the very first but then it can t pull the image from our ECR thread

William Grim · Answer

```During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 8 site packages prefect agent agent py line 391 in deploy flow run deployment info = self deploy flow flow run File usr local lib python3 8 site packages prefect agent docker agent py line 384 in deploy flow pull output = self docker client pull image stream=True decode=True File usr local lib python3 8 site packages docker api image py line 430 in pull self raise for status response File usr local lib python3 8 site packages docker api client py line 270 in raise for status raise create api error from http exception e File usr local lib python3 8 site packages docker errors py line 31 in create api error from http exception raise cls e response=response explanation=explanation docker errors APIError 500 Server Error for t 1 amazonaws com 2Fpath 2Fto 2Fimage Internal Server Error Head no basic auth credentials ```

William Grim · Answer

The `AWS ACCESS KEY ID` and `AWS SECRET ACCESS KEY` exist in the environment the agent is running in and in the location where I did flow registration from Is there something else to do to get `DockerRun` or docker agent to pull images from our private ECR

Anna Geller · Answer

So in the end you use a docker agent and Docker storage

Anna Geller · Answer

to authenticate your Docker agent with ECR you can run the command ```aws ecr get login password region region | docker login username AWS password stdin <http aws account id dkr ecr region amazonaws com|aws account id dkr ecr region amazonaws com>```

Anna Geller · Answer

you would need to run the above command in your terminal before you start your Docker agent This will update your Docker config file with the ECR credentials and will ensure that flow runs spun up as docker containers by your Docker agent will should be able to pull the image

William Grim · Answer

< Anna Geller> well we bring up a stack of docker containers and one of those containers is where we run a prefect agent it was running as a `local` agent but i thought that since it s not compatible with `DockerStorage` i need to switch to `DockerRun` and docker agent especially since our prod is k8s and we want to get prefect submitting k8s jobs instead of running local processes

Anna Geller · Answer

Note this caveat <https discourse prefect io t can i run a docker agent in a container 50|here> the docker agent must run as a local process not as a Docker container otherwise you ll get into trouble when scaling your flows The same with a local agent actually it should run as a local process and you can run it with supervisor to make it more robust and allowing the agent to restart itself

William Grim · Answer

oh yeah we bring up like a more full featured container and inside of that container we start a process that is the prefect agent

William Grim · Answer

is that what you mean

William Grim · Answer

oh i ll read what you posted

Anna Geller · Answer

but you then run docker in docker I d encourage you to check up the thread I shared and to run the agent as a local process to avoid issues

William Grim · Answer

a local process where on the host machine

Anna Geller · Answer

exactly on some VM where you would like to deploy it Alternative you spin up a separate KubernetesAgent e g in dev namespace for your non prod environment may actually get easier

William Grim · Answer

i m reading on that link that we should use `KubernetesAgent` if we want to have prefect inside the container i saw that `KubernetesRun` can have `image pull secrets` set i just wasn t sure if that kind of agent was going to work in docker

William Grim · Answer

yeah moving prefect outside docker and onto host machines is too big a dev change if we can use the k8s agent think that s possible in docker

Anna Geller · Answer

yes you re spot on in your use case it makes sense to use a `KubernetesAgent` for your dev environment as well you can pull secrets this way that s correct

William Grim · Answer

freakin awesome

William Grim · Answer

since we want to get prod to that end result that s nice i can try it in dev

Anna Geller · Answer

if you need an example for the Secret setup check this Readme and flow example in this repo <https github com anna geller packaging prefect flows >

William Grim · Answer

Thanks < Anna Geller> all your help the last couple days has been great I m gonna go through this and hopefully push us over the line

Anna Geller · Answer

sure keep us posted if you have any questions along the way

William Grim · Answer

< Christian Nuss>