As suggested by Anna, I've looked into her

dataflow-ops

repo and have tried to use the

ECSTask

from

prefect-aws

to circumvent the issue surrounding

DockerRegistry

for ECR, but I keep getting the error

KeyError: "No class found for dispatch key 'ecs-task' in registry for type 'Block'"

. Prefect==2.6.8 & prefect-aws==0.1.8

Hi Joshua! Do you have the

prefect-aws

package installed on the machine where your agent is running?

Thanks for the quick reply Ryan, do you mean the docker image of the agent?

yes, if you are running the agent in a container

That's likely my issue. Connecting with our DevOps team looks like we were just using the prefect image and not a custom image. Do all containers in our cluster need to be based off the custom image defined by:

FROM prefecthq/prefect:2-python3.9  # example base image 
RUN pip install s3fs prefect-aws
RUN prefect block register -m prefect_aws.ecs

or just the agent?

I believe the agent would need:

FROM prefecthq/prefect:2-python3.9  # example base image 
RUN pip install s3fs prefect-aws

The

prefect-aws

collection should auto-register ECSTask with the local install of Prefect, so I don't think you'll to do it again. The

prefect block register -m prefect_aws.ecs

command registers it with Prefect cloud or Orion server but it sounds like you've already done that. It's probably easiest to use the same image everywhere, but if you are using the default

S3

storage block I believe that for flow runs you would only need:

FROM prefecthq/prefect:2-python3.9  # example base image 
RUN pip install s3fs

If you go that route, double check which block you are using for flow storage;

prefect-aws

contains a storage block named

S3Bucket

that has a few more features than the

S3

block bundled with Prefect - but you'd know if you were using

S3Bucket

because you'd need to register

prefect_aws.s3

first. (Sorry for the long reply; just want to make sure I cover everything that might be useful)