Hi everyone! I am working on setting up my first flow on AWS infrastructure. Using Docker storage and pushing to AWS ECR, and running on ECS + Fargate. When I try to build and register my container I get this error:

[2021-05-28 09:59:27-0700] INFO - prefect.Docker | Pushing image to the registry...
Traceback (most recent call last):
  File "flow_registry.py", line 44, in <module>
    build_and_register_all_flows(all_flows)
  File "flow_registry.py", line 35, in build_and_register_all_flows
    storage = storage.build()
  File "/XXX/XXX/XXX/XXX/XXX/XXX/XXX/new_etl/venv/lib/python3.8/site-packages/prefect/storage/docker.py", line 303, in build
    self._build_image(push=push)
  File "/XXX/XXX/XXX/XXX/XXX/XXX/XXX/new_etl/venv/lib/python3.8/site-packages/prefect/storage/docker.py", line 378, in _build_image
    self.push_image(full_name, self.image_tag)
  File "/XXX/XXX/XXX/XXX/XXX/XXX/XXX/new_etl/venv/lib/python3.8/site-packages/prefect/storage/docker.py", line 586, in push_image
    raise InterruptedError(line.get("error"))
InterruptedError: denied: Your authorization token has expired. Reauthenticate and try again.

This is confusing to me, because I have double checked that my local AWS authentication is all setup properly in the environment. Is this referring to something else? Relatively new to Docker, ECR, and Prefect. Any help appreciated! Any ideas?

Hey @Jacob Goldberg, it’s happening with the

push_image

so this seems like an ECR thing. I’ll look for a way to test it

Thanks Kevin!

What happens if you do this in the command line:

aws ecr describe-repositories

?

I get an error

What is the error?

An error occurred (RepositoryNotFoundException) when calling the DescribeImages operation: The repository with name 'amazonlinux' does not exist in the registry with id 'XXXXX0963009'

oh i mean, replace the repository name with the one you want to pull an image from

haha whoops, my bad

Can you try

aws ecr get-authorization-token

in the command line and then check when it expires?

interesting it comes back with "expiresAt": "2021-05-28T222504.702000-07:00",

Yeah it looks fine. I think this is created through boto3 but let me take a look around

i am registering a number of flows to the same container so this is my setup:

...

all_flows = [test_flow_1, test_flow_2]


# define flow storage/config
STORAGE = Docker(registry_url='<http://XXXXX963009.dkr.ecr.us-east-1.amazonaws.com/|XXXXX963009.dkr.ecr.us-east-1.amazonaws.com/>',
                 image_name='cal_val_etl_flows',
                 image_tag='latest',
                 python_dependencies=required,
                 # copy all relevant files into docker container
                 files={os.path.dirname(os.path.dirname(os.path.realpath(__file__))): "."}
                   )

RUN_CONFIG = ECSRun(run_task_kwargs={'cluster': 'cal_val_etl_cluster'},
                    execution_role_arn='arn:aws:iam::XXXXX963009',
                    labels=['ecs', 'dev'])


def build_and_register_all_flows(flow_list, storage=STORAGE, run_config=RUN_CONFIG):
    # assign run config and storage to all flows
    for flow in flow_list:
        flow.run_config = run_config
        storage.add_flow(flow)
    # build the Docker Image
    storage = storage.build()

    for flow in flow_list:
        # Reassign the new storage object to each Flow
        flow.storage = storage
        # Register each flow without building a second time
        flow.register(project_name="cal_val_etl", build=False)


build_and_register_all_flows(all_flows)

Can you see if you get the same error with a simpler flow like:

from prefect import Flow, task
from prefect.storage import Docker

@task
def abc():
    return 1

with Flow("abc") as flow:
    abc()

storage = Docker(registry_url="<http://123454324543.dkr.ecr.us-east-2.amazonaws.com|123454324543.dkr.ecr.us-east-2.amazonaws.com>",
                 dockerfile="Dockerfile")

flow.storage = storage
flow.register("project_name")

I tried this sample with my own storage parameters:

STORAGE = Docker(registry_url='<http://XXXXXX963009.dkr.ecr.us-east-1.amazonaws.com/|XXXXXX963009.dkr.ecr.us-east-1.amazonaws.com/>',
                 image_name='cal_val_etl_flows',
                 image_tag='latest',
                 python_dependencies=required,
                 # copy all relevant files into docker container
                 files={os.path.dirname(os.path.dirname(os.path.realpath(__file__))): "."}
                   )

and i get the same error

Did you do something like

aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin <http://123456678.dkr.ecr.us-east-2.amazonaws.com|123456678.dkr.ecr.us-east-2.amazonaws.com>

i did initially (yesterday), but i guess i needed to run that again to refresh the token. That worked! thank you so much for your support @Kevin Kho

Ok that’s good that’s its working. I think there should be a setting in ECR to increase token duration

ok, i will look into that. thanks again