https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> What is best practice for defining EC...
# ask-marvin
s
@Marvin What is best practice for defining ECR images on a per-flow basis (Using ecs worker and, prefect.yaml configuration)
m
Here are the best practices for defining ECR images on a per-flow basis using ECS workers and 
prefect.yaml
configuration: 1. ECR Repository Setup: - Create your ECR repository using AWS CLI:
Copy code
sh
      aws ecr create-repository --repository-name <my-ecr-repo> --region <region>
2. Creating 
prefect.yaml
: - Define your ECR image and ECS worker in the 
prefect.yaml
file. Below is a template:
Copy code
yaml
    name: ecs-worker-guide
    prefect-version: 2.14.20

    build:
      - prefect_docker.deployments.steps.build_docker_image:
          id: build_image
          requires: prefect-docker>=0.3.1
          image_name: <my-ecr-repo>
          tag: latest
          dockerfile: auto

    push:
      - prefect_docker.deployments.steps.push_docker_image:
          requires: prefect-docker>=0.3.1
          image_name: '{{ build_image.image_name }}'
          tag: '{{ build_image.tag }}'

    deployments:
      - name: my_ecs_deployment
        entrypoint: flow.py:my_flow
        work_pool:
          name: ecs-dev-pool
        job_variables:
          image: '{{ build_image.image }}'
        pull:
          - prefect.deployments.steps.set_working_directory:
              directory: /opt/prefect/
3. Execution: - Use Prefect to build and push your defined Docker image:
Copy code
sh
      prefect deployment build -n my_ecs_deployment
- Deploy your flow to the ECS cluster:
Copy code
sh
      prefect deployment apply
These steps ensure that your ECS worker uses the specified ECR image for flow execution, allowing for scalable and reproducible deployments. For more details, refer to Prefect ECS Guide.
n
this is right except its 
prefect deploy
now
s
Ty @Nate. Question, is this pattern supposed to create a new ECS Task and run the “targeted” image. I guess I am having trouble with the distinction between the ECS service task (aka the task running the worker 
prefect worker start
) and the actual execution of my flow ( which ideally is running 
my_flow.py
via a different Docker container).
n
I'd say you pretty much already have the idea • the worker runs as a "service" in one container and polls for work • when it finds some, it spins up another container in ECS for the flow run are you trying to set the image for the flow run container or the worker container?
s
The former, I am trying to set the image for the flow run. I already have the worker container polling on ECS using a prefect base image (
prefecthq/prefect:2-python3.10
). But for the flow run, I have a custom image on ECR that contains the flow code I want to run.
n
that makes sense! yeah marvin's answer is generally how you use custom image for a given deployment (to use the lingo, a job variable override for the 
image
variable of your ECS work pool) if you dont want to have building/pushing the image as a part of the 
prefect deploy
, you can remove those 
build
and 
push
steps and hardcode a reference to your ECR image in the override or make a Variable that you reference there so you can change it out later without changing your 
prefect.yaml
s
Gotcha ty! This a bit more in the weeds but I’ve been running into the following issue when running the deployment.
Copy code
File "/usr/local/lib/python3.10/site-packages/prefect_aws/workers/ecs_worker.py", line 981, in _register_task_definition
    response = ecs_client.register_task_definition(**task_definition)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 1021, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images.
Which seems strange because 1. I’ve defined the task definition/execution role for the ecs service 2. Checked the permission policies on the execution role. I tried following the Prefect ECS Guide as close as i could. Any thoughts on where to look?
n
have you set your execution role ARN on your ecs work pool?
s
Is that referring to the step where you set the execution role on the task definition?
Copy code
{
    "family": "prefect-worker-task",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "512",
    "memory": "1024",
    "executionRoleArn": "<ecs-task-role-arn>", <--- HERE
    "taskRoleArn": "<ecs-task-role-arn>", <--- HERE
    "containerDefinitions": [
If so yes, if not, not sure I know where else I set the executeion role ARN. (also i’m using the same ARN for both execution + task for this scenario)
n
so that’s the execution role arn for the worker but from the error message it seems you need it for the flow run container when using fargate as a capacity provider (i’m not an ecs buff, but that’s what it seems like) on the ECS work pool you should find a field called execution role arn
s
ahhh and when you say “on the ECS work pool” is that done on the deployment in prefect.yaml? Not sure where that field or where I can look for a list of available fields. Been looking though python SDK api reference.
n
if you go in the UI to work pools in the left nav and go to Edit your existing ECS work pool, you should find this field
🎉 1
if you go into the Advanced JSON tab, you'll see the actual base job template and youll notice that execution_role_arn is in the list of 
variables
which means you can override that value in the 
job_variables
section of your deployment in the 
prefect.yaml
just like other things like 
image
so you can: • change it for all deployments that use that work pool by changing the value in the UI directly • or just for that deployment in the yaml with an override
s
Awesome, got it, yeh I did not set anything there. I’ll dive into that. I should be good for now. Appreciate all the help @Nate 🙌
n
catjam
8 Views
Open in Slack
PreviousNext
Powered by