does anyone have experience running a docker image from a private registry (Amazon ECR) and the script stored on AWS CodeCommit?

Hey, you need to do something like

aws ecr get-login-password

to authenticate the gent. See this under the

Create storage for flows with ECR

section. And then you would use CodeCommit for Storage (and provide the credentials to that) + ECR in the RunConfig. What have you tried so far?

I'm able to pull the image fine, but inside the image I am getting "no module named boto3"

Where are you running the agent?

here's my flow setup, maybe the argument is different for the dockerrun config to pass in credentials?

with Flow('Test DockerRun Configuration',
          run_config = DockerRun(
                  image = 'my_aws_image,
                  labels = ['sradocker'],
                  secrets=["AWS_CREDENTIALS"]
          ),
          storage = CodeCommit(
                  repo = 'advisorscodebase',
                  path = 'InvestmentManagement/test_docker.py',
                  commit = 'srockjoe',
                  secrets = ["AWS_CREDENTIALS"],
                  client_options = { "region_name":"us-east-2" }
          )
          ) as flow:

Yeah I think you want to pass them through the env in DockerRun

flow.run_config = DockerRun(env={"SOME_VAR": "value"})

The agent is running on the server machine, authenticated and piped to the docker daemon. It says it's able to successfully pull the image. Looks like the error is coming inside the image

Wait no that won’t fix the issue. I think you need boto3 in your image?

i thought so too but I installed prefect[aws

Did you use conda? Conda doesn’t respect extras

hell no haha I can't stand conda

Do you have more on the traceback? What executor are you using?

just the default executor. Here's the whole traceback:

[2021-07-06 19:31:32,919] INFO - agent | Deploying flow run 5b13f589-39d9-483f-89f9-9c4f8dc81ea5 to execution environment...
[2021-07-06 19:31:33,082] INFO - agent | Pulling image <http://393312900658.dkr.ecr.us-east-2.amazonaws.com/py38_prefect:latest|393312900658.dkr.ecr.us-east-2.amazonaws.com/py38_prefect:latest>...
[2021-07-06 19:31:33,330] INFO - agent | Successfully pulled image <http://393312900658.dkr.ecr.us-east-2.amazonaws.com/py38_prefect:latest|393312900658.dkr.ecr.us-east-2.amazonaws.com/py38_prefect:latest>
[2021-07-06 19:31:34,722] INFO - agent | Completed deployment of flow run 5b13f589-39d9-483f-89f9-9c4f8dc81ea5
**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.10.1
** Copyright (c) 2021 Microsoft Corporation
**********************************************************************
No module named 'boto3'
c:\program files (x86)\python38-32\lib\site-packages\prefect\client\client.py:170: UserWarning: Client was created with an API token configured for authentication. API tokens are deprecated, please use API keys instead.
  warnings.warn(
Traceback (most recent call last):
  File "c:\program files (x86)\python38-32\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\program files (x86)\python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Program Files (x86)\Python38-32\Scripts\prefect.exe\__main__.py", line 7, in <module>
  File "c:\program files (x86)\python38-32\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\program files (x86)\python38-32\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\program files (x86)\python38-32\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\program files (x86)\python38-32\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\program files (x86)\python38-32\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\program files (x86)\python38-32\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\program files (x86)\python38-32\lib\site-packages\prefect\cli\execute.py", line 96, in flow_run
    raise exc
  File "c:\program files (x86)\python38-32\lib\site-packages\prefect\cli\execute.py", line 73, in flow_run
    flow = storage.get_flow(flow_data.name)
  File "c:\program files (x86)\python38-32\lib\site-packages\prefect\storage\codecommit.py", line 70, in get_flow
    client = self._boto3_client
  File "c:\program files (x86)\python38-32\lib\site-packages\prefect\storage\codecommit.py", line 117, in _boto3_client
    from prefect.utilities.aws import get_boto_client
  File "c:\program files (x86)\python38-32\lib\site-packages\prefect\utilities\aws.py", line 10, in <module>
    import boto3
ModuleNotFoundError: No module named 'boto3'

Are you using environments inside your container?

no sir

Are you sure you’re pulling the latest container on the ECS side?

It's not an ECS like one of the amazon-owned ones, it's an image I built and stored on ECR, but yeah I'm definitely pulling the latest n I just rebuilt it and tried again to no avail

I imagine that error would be different though, This is the import of boto3 failing

if it's a docker run I should be using a docker agent I thought?

That’s right. Just wanted to be sure

It just opens up command prompt when you run the container. Yeah good idea I can try that

Oh no worries! Glad you got it sorted

thanks again dude!