https://prefect.io logo
Title
j

Joshua Grant

11/18/2022, 5:30 PM
As suggested by Anna, I've looked into her
dataflow-ops
repo and have tried to use the
ECSTask
from
prefect-aws
to circumvent the issue surrounding
DockerRegistry
for ECR, but I keep getting the error
KeyError: "No class found for dispatch key 'ecs-task' in registry for type 'Block'"
. Prefect==2.6.8 & prefect-aws==0.1.8
1
ecs-task
is the name of my
ECSTask
r

Ryan Peden

11/18/2022, 5:38 PM
Hi Joshua! Do you have the
prefect-aws
package installed on the machine where your agent is running?
(it sounds like you probably do, but just want to check on that first before looking at other possible causes)
j

Joshua Grant

11/18/2022, 5:40 PM
Thanks for the quick reply Ryan, do you mean the docker image of the agent?
r

Ryan Peden

11/18/2022, 5:41 PM
yes, if you are running the agent in a container
j

Joshua Grant

11/18/2022, 5:43 PM
That's likely my issue. Connecting with our DevOps team looks like we were just using the prefect image and not a custom image. Do all containers in our cluster need to be based off the custom image defined by:
FROM prefecthq/prefect:2-python3.9  # example base image 
RUN pip install s3fs prefect-aws
RUN prefect block register -m prefect_aws.ecs
or just the agent?
r

Ryan Peden

11/18/2022, 6:00 PM
I believe the agent would need:
FROM prefecthq/prefect:2-python3.9  # example base image 
RUN pip install s3fs prefect-aws
The
prefect-aws
collection should auto-register ECSTask with the local install of Prefect, so I don't think you'll to do it again. The
prefect block register -m prefect_aws.ecs
command registers it with Prefect cloud or Orion server but it sounds like you've already done that. It's probably easiest to use the same image everywhere, but if you are using the default
S3
storage block I believe that for flow runs you would only need:
FROM prefecthq/prefect:2-python3.9  # example base image 
RUN pip install s3fs
If you go that route, double check which block you are using for flow storage;
prefect-aws
contains a storage block named
S3Bucket
that has a few more features than the
S3
block bundled with Prefect - but you'd know if you were using
S3Bucket
because you'd need to register
prefect_aws.s3
first. (Sorry for the long reply; just want to make sure I cover everything that might be useful)
:gratitude-thank-you: 1
🌮 1