https://prefect.io logo
Channels
Powered by
#prefect-community
Title
# prefect-community
j

Jason

04/22/2022, 4:47 PM
Where does the Docker storage inherit its credentials from? I assume boto3, but I'm getting a weird behavior on Github Actions. Using the manual docker build and push with the ECR Login action (https://github.com/aws-actions/amazon-ecr-login#usage), I can push my image just fine to ECR. However, when I run this with flow.register() it fails on no authorization.
Copy code
raise InterruptedError(line.get("error"))
      InterruptedError: no basic auth credentials
Well, actually then the error would also occur locally but it works. OK, now I'm stumped again
k

Kevin Kho

04/22/2022, 5:01 PM
Are you doing the
Copy code
aws ecr get-login-password
?
j

Jason

04/22/2022, 5:43 PM
Yup
Even tried with a manual login instead of the Github Action, inlining in the prefect register step in GA. Still failed. Odd
k

Kevin Kho

04/22/2022, 5:44 PM
This login only lasts 12 hours right?
j

Jason

04/22/2022, 5:46 PM
Sure, but each run it gets renewed.
Since each run triggers the ecr-login in gha
k

Kevin Kho

04/22/2022, 5:48 PM
Let me answer the first question though. Docker storage basically will work if 
docker push
works
j

Jason

04/22/2022, 5:55 PM
That's not what I'm seeing. I just tagged and pushed via docker push. And using the same environment, registering the flow fails with no basic auth
k

Kevin Kho

04/22/2022, 6:00 PM
That’s pretty weird. You can find the code here, it should but be making a Docker client and pushing. That Docker Client should just be using Docker in the same environment. Maybe you can try using the Python 
client.push()
and if it fails, we can look into why 
docker-py
isn’t getting the credentials
j

Jason

04/22/2022, 6:03 PM
Copy code
AWS_PROFILE=github make docker-login
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <http://EDITED.dkr.ecr.us-east-1.amazonaws.com|EDITED.dkr.ecr.us-east-1.amazonaws.com>
WARNING! Your password will be stored unencrypted in /home/EDITED/.docker/config.json.
Configure a credential helper to remove this warning. See
<https://docs.docker.com/engine/reference/commandline/login/#credentials-store>
Login Succeeded
Copy code
Successfully tagged <http://EDITED.DKR.ECR.US-EAST-1.AMAZONAWS.COM/EDITED-prod-platform-prefect:latest|EDITED.DKR.ECR.US-EAST-1.AMAZONAWS.COM/EDITED-prod-platform-prefect:latest>
[2022-04-22 13:03:59-0500] INFO - prefect.Docker | Pushing image to the registry...
    Error building storage:
      Traceback (most recent call last):
        File "/home/EDITED/.local/share/virtualenvs/data-vrc-T2jvyYmD/lib/python3.9/site-packages/prefect/cli/build_register.py", line 457, in build_and_register
    storage.build()
        File "/home/EDITED/.local/share/virtualenvs/data-vrc-T2jvyYmD/lib/python3.9/site-packages/prefect/storage/docker.py", line 325, in build
    self._build_image(push=push)
        File "/home/EDITED/.local/share/virtualenvs/data-vrc-T2jvyYmD/lib/python3.9/site-packages/prefect/storage/docker.py", line 399, in _build_image
    self.push_image(full_name, self.image_tag)
        File "/home/EDITED/.local/share/virtualenvs/data-vrc-T2jvyYmD/lib/python3.9/site-packages/prefect/storage/docker.py", line 613, in push_image
    raise InterruptedError(line.get("error"))
      InterruptedError: no basic auth credentials

  Registering 'hello-flow'... Error
================== 0 registered, 1 errored ==================
k

Kevin Kho

04/22/2022, 6:16 PM
I can’t really find anything about why 
docker-py
isn’t picking it up. This is a widely used setup among Prefect users though so this is very weird
j

Jason

04/22/2022, 6:30 PM
Yeah - tried a few more things but ended up replicating. I can docker push fine in my terminal after ecr-login, but I can't do a prefect register --project .. --path '*. to register the flow
@Kevin Kho: i think it's related to the Linux credential store. The flow registers fine on MacOS, which uses the MacOS keychain. But on Github actions (ubuntu) and my desktop (ubuntu), docker-py gives the same error.
k

Kevin Kho

04/22/2022, 7:02 PM
Man not sure I can advise on that. Do you have multiple log-ins or something? I feel like I have seen people deploy with Github actions though
a

Anna Geller

04/22/2022, 7:04 PM
it's worth opening an issue, I see this error the second time today @Marvin open "Flow registration with Docker storage can't push the image to ECR due to `InterruptedError: no basic auth credentials`" @Jason Motley as a workaround, for now, you could use the solution from the Discourse topic I linked, i.e. manually pushing the image yourself to ECR
m

Marvin

04/22/2022, 7:04 PM
j

Jason

04/22/2022, 7:09 PM
There hasn't been a release since 2021, so I don't think the issue is docker-py, per se then: https://github.com/docker/docker-py/releases/tag/5.0.3
Is there anybody that does have a GA ECR push that would share their workflow, alternatively? Perhaps I'm setting it up incorrectly.
k

Kevin Kho

04/22/2022, 7:11 PM
Let’s look for one around here
a

Anna Geller

04/22/2022, 7:12 PM
you said it works when you push manually, but doesn't work when you do that with flow.register()? if not, please clarify your issue a bit more Jason
👍 1
k

Kevin Kho

04/22/2022, 7:12 PM
But not ECR. Will continue looking
j

Jason

04/22/2022, 7:13 PM
That's correct. I suppose I can just add a step in GHA to tag and push the image, and have .register() use the same tag and it should just pull it
k

Kevin Kho

04/22/2022, 7:14 PM
That’s true
17 Views
Powered by