https://prefect.io logo
Join Slack
Powered by
Alright, I'm slowly pushing further with `DockerSt...
# prefect-community
w
Alright, I'm slowly pushing further with 
DockerStorage
and then realized I can't use a 
local
agent to run flows that use that. No problem... since my local agent is setup in a docker container, I figured I would just use a 
DockerRun
config. This seems to go alright at the very first, but then it can't pull the image from our ECR. 🧵
Copy code
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/agent.py", line 391, in _deploy_flow_run
    deployment_info = self.deploy_flow(flow_run)
  File "/usr/local/lib/python3.8/site-packages/prefect/agent/docker/agent.py", line 384, in deploy_flow
    pull_output = self.docker_client.pull(image, stream=True, decode=True)
  File "/usr/local/lib/python3.8/site-packages/docker/api/image.py", line 430, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=my_account_id.dkr.ecr.us-wes>
t-1.amazonaws.com%2Fpath%2Fto%2Fimage: Internal Server Error ("Head "<https://my_account_id.dkr.ecr.us-west-1.amazo>
<http://naws.com/v2/path/to/image/manifests/latest|naws.com/v2/path/to/image/manifests/latest>": no basic auth credentials")
The 
AWS_ACCESS_KEY_ID
and 
AWS_SECRET_ACCESS_KEY
exist in the environment the agent is running in and in the location where I did flow registration from. Is there something else to do to get 
DockerRun
or docker agent to pull images from our private ECR?
a
So in the end you use a docker agent and Docker storage?
to authenticate your Docker agent with ECR, you can run the command:
Copy code
aws ecr get-login-password --region region | docker login --username AWS --password-stdin <http://aws_account_id.dkr.ecr.region.amazonaws.com|aws_account_id.dkr.ecr.region.amazonaws.com>
you would need to run the above command in your terminal before you start your Docker agent. This will update your Docker config file with the ECR credentials and will ensure that flow runs spun up as docker containers by your Docker agent will (should) be able to pull the image
w
@Anna Geller well, we bring up a stack of docker containers, and one of those containers is where we run a prefect agent. it was running as a 
local
agent, but i thought that since it's not compatible with 
DockerStorage
, i need to switch to 
DockerRun
and docker agent, especially since our prod is k8s and we want to get prefect submitting k8s jobs instead of running local processes.
a
Note this caveat here - the docker agent must run as a local process, not as a Docker container, otherwise you’ll get into trouble when scaling your flows. The same with a local agent actually, it should run as a local process and you can run it with supervisor to make it more “robust” and allowing the agent to restart itself
w
oh yeah, we bring up like a more full-featured container, and inside of that container, we start a process that is the prefect agent
is that what you mean?
oh, i'll read what you posted
a
but you then run docker in docker, I’d encourage you to check up the thread I shared and to run the agent as a local process to avoid issues
w
a local process where? on the host machine?
a
exactly, on some VM where you would like to deploy it. Alternative: you spin up a separate KubernetesAgent e.g. in “dev” namespace for your non-prod environment - may actually get easier
w
i'm reading on that link that we should use 
KubernetesAgent
if we want to have prefect inside the container. i saw that 
KubernetesRun
can have 
image_pull_secrets
set. i just wasn't sure if that kind of agent was going to work in docker
yeah, moving prefect outside docker and onto host machines is too big a dev change if we can use the k8s agent. think that's possible in docker?
a
yes, you’re spot on, in your use case it makes sense to use a 
KubernetesAgent
for your dev environment as well - you can pull secrets this way, that’s correct.
w
freakin' awesome
since we want to get prod to that end result, that's nice i can try it in dev
a
if you need an example for the Secret setup, check this Readme and flow example in this repo https://github.com/anna-geller/packaging-prefect-flows/
w
Thanks @Anna Geller; all your help the last couple days has been great. I'm gonna go through this and hopefully push us over the line.
a
sure, keep us posted if you have any questions along the way
đź‘Ť 1
w
@Christian Nuss
4 Views
Open in Slack
PreviousNext
Powered by