https://prefect.io logo
Title
j

Joshua Grant

11/16/2022, 3:22 PM
Has anyone had any luck creating a
DockerRegistry
block with AWS ECR?
โœ… 1
d

Dev Dabke

11/16/2022, 3:58 PM
I am also working on this!
j

Joshua Grant

11/16/2022, 3:59 PM
I was following this issue on github: https://github.com/PrefectHQ/prefect/issues/6230, which brought be to this PR: https://github.com/PrefectHQ/prefect-aws/pull/68, but was wondering if there was a workaround for now.
a

Anna Geller

11/16/2022, 8:07 PM
it's on our radar, afaik @alex team has it on their backlog until then I recommend using ECSTask with ECR being part of it natively - we have a repo +. blog post + video here https://github.com/anna-geller/dataflow-ops
:upvote: 1
๐Ÿ™Œ 1
n

Nikhil Jain

12/13/2022, 9:00 PM
@Anna Geller what storage type / block should I use with ECSTask when my flow code is in a ECR image?
a

Anna Geller

12/13/2022, 9:07 PM
hahaha
are you trying to see if me or Marvin gives a better answer? ๐Ÿ˜‚
n

Nikhil Jain

12/13/2022, 9:08 PM
I donโ€™t think Marvin can beat you anytime soon! ๐Ÿ™‚ I just thought you might be on vacation (since you were not online) so I tried asking marvin as well. ๐Ÿ™‚
๐Ÿ™Œ 1
a

Anna Geller

12/13/2022, 9:10 PM
the right and easiest solution to this would likely be to point the path to the path on the image so e.g. on your deployment, e.g. if on the image myflow.py is baked into this location /opt/prefect/flows/myflow.py, then this should work:
prefect deployment build -n dev -a -ib ecs-task/dev myflow.py:myflow --path /opt/prefect/flows --skip-upload
n

Nikhil Jain

12/13/2022, 9:14 PM
Thanks, couple of followup questions: 1. if I supply a
task_definition_arn
to my ECSTask block, which all of the settings in the task_definition be applied automatically? or do I have to supply some of the settings in the ECSTask block settings? 2. e.g. where should I add the ECR image url? in the ECSTask block? or in the
task_definition
(which I am creating using terraform).
a

Anna Geller

12/13/2022, 9:20 PM
Good question and one for which Andrew has recently added a PR to clarify https://github.com/PrefectHQ/prefect-aws/commit/cdd40bdacec5465050e3fc4f174f4301db294ebc if you register a task definition yourself, you set the image on the task definition if you let Prefect create the task definition for you, you specify the image on the ECSTask block
๐Ÿ™Œ 2
n

Nikhil Jain

12/13/2022, 9:24 PM
How does the agent know what
command
to run in the ECS Task? Do I have to hardcode the command in the ECR image (or in the task-definition)? Or will the agent automatically add a
command
override when creating the task?
a

Anna Geller

12/13/2022, 9:26 PM
another good question and I stumbled upon it myself lately. No need to do that. Even if your image would have say entrypoint like:
ENTRYPOINT ["prefect", "agent", "start", "-q", "default"]
any Infrastructure block by default overrides it to
["python", "-m", "prefect.engine"]
๐Ÿ™ 1
๐Ÿค” 1
n

Nikhil Jain

12/13/2022, 9:54 PM
I made some progress, however now getting an unexpected error in the
agent
logs:
File "/usr/local/lib/python3.10/site-packages/prefect_aws/ecs.py", line 693, in _create_task_and_wait_for_start
    network_config = self._load_vpc_network_config(self.vpc_id, boto_session)
  File "/usr/local/lib/python3.10/site-packages/prefect_aws/ecs.py", line 1228, in _load_vpc_network_config
    vpcs = ec2_client.describe_vpcs(**describe)["Vpcs"]
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnauthorizedOperation) when calling the DescribeVpcs operation: You are not authorized to perform this operation.
My flow is setup to run on FARGATE. I noticed that this stack trace is using
ec2_client
. Is this expected? The
task_execution_role
for my agent does not have any
ec2:*
capabilities, and used to work fine on Prefect1 with the same network config.
a

Anna Geller

12/13/2022, 9:55 PM
I'd need to see the full block config
generally, this is needed to start a task because even if you have a task definition, you need to run it in a specific subnet and to do that we check VPC network config unless you specify the subnets explicitly (I think)
n

Nikhil Jain

12/13/2022, 10:17 PM
ah.. thatโ€™s very helpful! I need to add the subnets and vpc-id as task_customizations.
๐Ÿ‘ 1
๐Ÿ™Œ 1
I was able to get my flows working on ECS! Couple of additional notes for anyone trying to do this: 1. The name of the ECS task container in which the flow is supposed to be run should be โ€œprefectโ€. 2. I added vpc_id in the ECSTask params, and aded subnet_ids and security_group_ids in
networkConfiguration
as task_customizations. 3. turns out ECSTask calls
ec2_client.describe_vpcs
even when vpc_id is provided explicitly. So had to add that policy to the task_execution_role.
:party-parrot: 1
a

Anna Geller

12/14/2022, 4:26 AM
Thanks sooo much and nice work! Interesting find about describe vpc, we'll need to cross check
๐Ÿ‘ 1