Quick question regarding the Docker Agent I have a VM on whi

Question

Quick question regarding the Docker Agent I have a VM on which I run Prefect Server + a Docker Agent Since I am using DockerRun I need to pull the image from a registry I have all authorizations properly set up doing `docker pull myregistry myimage latest` from the VM works flawlessly yet I can t run my flows since I get a 500 ```500 Server Error for Internal Server Error Head unauthorized authentication required visit for more information ``` My registry is on Azure Container Registries and from the VM I am properly logged in and can pull the image Does the agent need some sort of additional authentication

Kevin Kho · Answer

Hey < Pierre Monico> your setup sounds right to me Wouldn t you get an Auth error has opposed to the Internal Server Error Is your agent running in a Docker container by chance

Pierre Monico · Answer

Agreed it s confusing because it spits a 500 but down the road it mentions some authorization issue with ACR My agent simply runs with `prefect agent docker start` I m going to try a few more things today and post it here but I really have no clue what s going on

Pierre Monico · Answer

So I assumed the faulty call is happening at <https github com PrefectHQ prefect blob master src prefect agent docker agent py L384|this line> I tried to reproduce it directly on my VM where `prefect agent docker start` is run by writing a short Python script doing the same thing and it works So from the VM I can `docker pull` I can `docker client pull ` in Python but somehow when done from the agent it s not working I tried so many things that I am not sure what to do next any clues

Pierre Monico · Answer

I also tried to write a script that replicates <https github com PrefectHQ prefect blob master src prefect agent docker agent py L471|this line> but it also works without any issues

Pierre Monico · Answer

Is there any place where I can see the actual agent logs like Python exceptions not the flow runs

Kevin Kho · Answer

I think you can if you do `prefect agent docker start log level=DEBUG` and this will give you debug level logs for the agent What a weird situation since the `pull ` should be equivalent to the `docker pull` for the most part

Pierre Monico · Answer

Ok thanks Yes especially when I can write the same lines in Python and even those work on the machine

Pierre Monico · Answer

Can t wait to finally fix it I have the feeling that it will somehow an error on my end haha

Pierre Monico · Answer

Ok as expected I am actually running the prefect server + agent as `systemd` service that I wrote but I did not include the `User=` clause gt gt it was running as `root` but only my current user was in the `docker` group and authenticated would have thought that `root` always is but oh well I finally solved it

Pierre Monico · Answer

Thanks < Kevin Kho> again for the help

Kevin Kho · Answer

Wow What a find

Pierre Monico · Answer

Should have probably not written the `systemd` service before I actually got all my devops pipelines right smile `premature optimization is the root of all evil` holds true again

Pierre Monico · Answer

On a side note as far as I can tell even if you run `prefect server start` as a non root user some services and especially the one pulling from docker will still be run as `root` Effectively that means you need to make sure that root is authenticated to the registry you are trying to pull the image from there is no way of just having a non root user do that

Pierre Monico · Answer

<https github com PrefectHQ prefect issues 2824|This> might be related

Kevin Kho · Answer

I don t know much about that Do you think the fix is easy

Pierre Monico · Answer

I am not too sure but actually I hit the problem again Now both `root` and the user running `prefect start server` are authenticated to the registry I can both `docker pull` and `sudo docker pull` the images successfully but when running from the server agent I get the original error again

Kevin Kho · Answer

Ah you can chime in on that issue there