Quick question regarding the Docker Agent: I have ...
# prefect-server
Quick question regarding the Docker Agent: I have a VM on which I run Prefect Server + a Docker Agent. Since I am using DockerRun, I need to pull the image from a registry. I have all authorizations properly set up (doing
docker pull myregistry/myimage:latest
from the VM works flawlessly), yet I can’t run my flows since I get a 500:
Copy code
500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=myregistry%2Fmyimage>: Internal Server Error ("Head <https://myregistry/v2/myimage/manifests/latest>: unauthorized: authentication required, visit <https://aka.ms/acr/authorization> for more information.")
My registry is on Azure Container Registries and from the VM I am properly logged in and can pull the image. Does the agent need some sort of additional authentication?
Hey @Pierre Monico, your setup sounds right to me. Wouldn’t you get an Auth error has opposed to the Internal Server Error? Is your agent running in a Docker container by chance?
Agreed, it’s confusing because it spits a 500 but down the road it mentions some authorization issue with ACR. My agent simply runs with
prefect agent docker start
. I’m going to try a few more things today and post it here but I really have no clue what’s going on.
So I assumed the faulty call is happening at this line. I tried to reproduce it directly on my VM (where
prefect agent docker start
is run) by writing a short Python script doing the same thing - and it works. So from the VM I can
docker pull
, I can
(in Python), but somehow when done from the agent it’s not working… I tried so many things that I am not sure what to do next… any clues?
I also tried to write a script that replicates this line, but it also works without any issues.
Is there any place where I can see the actual agent logs - like Python exceptions, not the flow runs?
I think you can if you do
prefect agent docker start --log-level=DEBUG
and this will give you debug level logs for the agent. What a weird situation since the
should be equivalent to the
docker pull
for the most part.
Ok thanks! Yes, especially when I can write the same lines in Python and even those work on the machine.
Can’t wait to finally fix it; I have the feeling that it will somehow an error on my end haha
Ok as expected: I am actually running the prefect server + agent as
service (that I wrote), but I did not include the
clause >> it was running as
, but only my current user was in the
group and authenticated… would have thought that
always is, but oh well, I finally solved it!
Thanks @Kevin Kho again for the help!!
Wow! What a find!
Should have probably not written the
service before I actually got all my (devops) pipelines right 😄
premature optimization is the root of all evil
holds true again
On a side note: as far as I can tell, even if you run
prefect server start
as a non-root user, some services (and especially the one pulling from docker) will still be run as
. Effectively that means you need to make sure that root is authenticated to the registry you are trying to pull the image from - there is no way of just having a non-root user do that.
This might be related.
I don’t know much about that. Do you think the fix is easy?
I am not too sure but actually I hit the problem again. Now both
and the user running
prefect start server
are authenticated to the registry (I can both
docker pull
sudo docker pull
the images successfully) but when running from the server/agent I get the original error again…
Ah you can chime in on that issue there.