https://prefect.io logo
Join Slack
Powered by
Has anyone had any luck creating a `DockerRegistry...
# ask-community
j
Has anyone had any luck creating a 
DockerRegistry
block with AWS ECR?
✅ 1
d
I am also working on this!
j
I was following this issue on github: https://github.com/PrefectHQ/prefect/issues/6230, which brought be to this PR: https://github.com/PrefectHQ/prefect-aws/pull/68, but was wondering if there was a workaround for now.
a
it's on our radar, afaik @alex team has it on their backlog until then I recommend using ECSTask with ECR being part of it natively - we have a repo +. blog post + video here https://github.com/anna-geller/dataflow-ops
upvote 1
🙌 1
n
@Anna Geller what storage type / block should I use with ECSTask when my flow code is in a ECR image?
a
hahaha
are you trying to see if me or Marvin gives a better answer? 😂
n
I don’t think Marvin can beat you anytime soon! 🙂 I just thought you might be on vacation (since you were not online) so I tried asking marvin as well. 🙂
🙌 1
a
the right and easiest solution to this would likely be to point the path to the path on the image so e.g. on your deployment, e.g. if on the image myflow.py is baked into this location /opt/prefect/flows/myflow.py, then this should work:
Copy code
prefect deployment build -n dev -a -ib ecs-task/dev myflow.py:myflow --path /opt/prefect/flows --skip-upload
n
Thanks, couple of followup questions: 1. if I supply a 
task_definition_arn
to my ECSTask block, which all of the settings in the task_definition be applied automatically? or do I have to supply some of the settings in the ECSTask block settings? 2. e.g. where should I add the ECR image url? in the ECSTask block? or in the 
task_definition
(which I am creating using terraform).
a
Good question and one for which Andrew has recently added a PR to clarify https://github.com/PrefectHQ/prefect-aws/commit/cdd40bdacec5465050e3fc4f174f4301db294ebc if you register a task definition yourself, you set the image on the task definition if you let Prefect create the task definition for you, you specify the image on the ECSTask block
🙌 2
n
How does the agent know what 
command
to run in the ECS Task? Do I have to hardcode the command in the ECR image (or in the task-definition)? Or will the agent automatically add a 
command
override when creating the task?
a
another good question and I stumbled upon it myself lately. No need to do that. Even if your image would have say entrypoint like:
Copy code
ENTRYPOINT ["prefect", "agent", "start", "-q", "default"]
any Infrastructure block by default overrides it to
Copy code
["python", "-m", "prefect.engine"]
🙏 1
🤔 1
n
I made some progress, however now getting an unexpected error in the 
agent
logs:
Copy code
File "/usr/local/lib/python3.10/site-packages/prefect_aws/ecs.py", line 693, in _create_task_and_wait_for_start
    network_config = self._load_vpc_network_config(self.vpc_id, boto_session)
  File "/usr/local/lib/python3.10/site-packages/prefect_aws/ecs.py", line 1228, in _load_vpc_network_config
    vpcs = ec2_client.describe_vpcs(**describe)["Vpcs"]
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnauthorizedOperation) when calling the DescribeVpcs operation: You are not authorized to perform this operation.
My flow is setup to run on FARGATE. I noticed that this stack trace is using 
ec2_client
. Is this expected? The 
task_execution_role
for my agent does not have any 
ec2:*
capabilities, and used to work fine on Prefect1 with the same network config.
a
I'd need to see the full block config
generally, this is needed to start a task because even if you have a task definition, you need to run it in a specific subnet and to do that we check VPC network config unless you specify the subnets explicitly (I think)
n
ah.. that’s very helpful! I need to add the subnets and vpc-id as task_customizations.
👍 1
🙌 1
I was able to get my flows working on ECS! Couple of additional notes for anyone trying to do this: 1. The name of the ECS task container in which the flow is supposed to be run should be “prefect”. 2. I added vpc_id in the ECSTask params, and aded subnet_ids and security_group_ids in 
networkConfiguration
as task_customizations. 3. turns out ECSTask calls 
ec2_client.describe_vpcs
even when vpc_id is provided explicitly. So had to add that policy to the task_execution_role.
🦜 1
a
Thanks sooo much and nice work! Interesting find about describe vpc, we'll need to cross check
👍 1
3 Views
Open in Slack
PreviousNext
Powered by