https://prefect.io logo
Join Slack
Powered by
Quick question regarding the Docker Agent: I have ...
# prefect-server
p
Quick question regarding the Docker Agent: I have a VM on which I run Prefect Server + a Docker Agent. Since I am using DockerRun, I need to pull the image from a registry. I have all authorizations properly set up (doing 
docker pull myregistry/myimage:latest
from the VM works flawlessly), yet I can’t run my flows since I get a 500:
Copy code
500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=myregistry%2Fmyimage>: Internal Server Error ("Head <https://myregistry/v2/myimage/manifests/latest>: unauthorized: authentication required, visit <https://aka.ms/acr/authorization> for more information.")
My registry is on Azure Container Registries and from the VM I am properly logged in and can pull the image. Does the agent need some sort of additional authentication?
k
Hey @Pierre Monico, your setup sounds right to me. Wouldn’t you get an Auth error has opposed to the Internal Server Error? Is your agent running in a Docker container by chance?
p
Agreed, it’s confusing because it spits a 500 but down the road it mentions some authorization issue with ACR. My agent simply runs with 
prefect agent docker start
. I’m going to try a few more things today and post it here but I really have no clue what’s going on.
So I assumed the faulty call is happening at this line. I tried to reproduce it directly on my VM (where 
prefect agent docker start
is run) by writing a short Python script doing the same thing - and it works. So from the VM I can 
docker pull
, I can 
docker_client.pull()
(in Python), but somehow when done from the agent it’s not working… I tried so many things that I am not sure what to do next… any clues?
I also tried to write a script that replicates this line, but it also works without any issues.
Is there any place where I can see the actual agent logs - like Python exceptions, not the flow runs?
k
I think you can if you do 
prefect agent docker start --log-level=DEBUG
and this will give you debug level logs for the agent. What a weird situation since the 
pull()
should be equivalent to the 
docker pull
for the most part.
p
Ok thanks! Yes, especially when I can write the same lines in Python and even those work on the machine.
Can’t wait to finally fix it; I have the feeling that it will somehow an error on my end haha
Ok as expected: I am actually running the prefect server + agent as 
systemd
service (that I wrote), but I did not include the 
User=
clause >> it was running as 
root
, but only my current user was in the 
docker
group and authenticated… would have thought that 
root
always is, but oh well, I finally solved it!
Thanks @Kevin Kho again for the help!!
k
Wow! What a find!
p
Should have probably not written the 
systemd
service before I actually got all my (devops) pipelines right 😄 
premature optimization is the root of all evil
holds true again
On a side note: as far as I can tell, even if you run 
prefect server start
as a non-root user, some services (and especially the one pulling from docker) will still be run as 
root
. Effectively that means you need to make sure that root is authenticated to the registry you are trying to pull the image from - there is no way of just having a non-root user do that.
This might be related.
k
I don’t know much about that. Do you think the fix is easy?
p
I am not too sure but actually I hit the problem again. Now both 
root
and the user running 
prefect start server
are authenticated to the registry (I can both 
docker pull
and 
sudo docker pull
the images successfully) but when running from the server/agent I get the original error again…
k
Ah you can chime in on that issue there.
3 Views
Open in Slack
PreviousNext
Powered by