Hey! A question about using a `task_definition_arn` in an `ECSTask` block, details below

Claire Herdeman

11/15/2022, 9:11 PM

Hey! A question about using a

task_definition_arn

in an

ECSTask

block, details below

✅ 1

Claire Herdeman

11/15/2022, 9:11 PM

Here's my current task definition as defined in terraform:

Copy code

resource "aws_ecs_task_definition" "prefect2Flow" {
  family = "prefect2Flow_${var.env}"
  requires_compatibilities =  ["FARGATE"]
  network_mode = "awsvpc"
  cpu = "512"
  memory = "1024"
  task_role_arn = aws_iam_role.prefect_flow_ecs_task_role.arn
  execution_role_arn = aws_iam_role.prefect_flow_ecs_execution_role.arn
  container_definitions = <<TASK_DEFINITION
  [
    {
        "name": "prefect",
        "image": "${aws_ecr_repository.integrations.repository_url}:latest",
        "environment": [{"name": "PREFECT_DEBUG_MODE", "value": "1"}],
        "logConfiguration": {
            "logDriver": "awslogs",
            "options": {
                "awslogs-group": "${aws_cloudwatch_log_group.prefect_flow.name}",
                "awslogs-region": "us-east-2",
                "awslogs-stream-prefix": "prefect_flow"
            }
        }
    }
  ]
  TASK_DEFINITION
}

Claire Herdeman

11/15/2022, 9:12 PM

And the ECSTask:

Copy code

ecs = ECSTask(
        task_definition_arn=task,
        cluster=cluster,
        network_mode="awsvpc",
        task_customizations=[
            {
                "op": "replace",
                "path": f"{NETWORK_PATH}/assignPublicIp",
                "value": "DISABLED",
            },
            {
                "op": "add",
                "path": f"{NETWORK_PATH}/subnets",
                "value": get_networking(env, "Subnets"),
            },
            {
                "op": "add",
                "path": f"{NETWORK_PATH}/securityGroups",
                "value": get_networking(env, "SecurityGroups"),
            }
        ]
    )
    ecs.save(f"{env}", overwrite=True)

Claire Herdeman

11/15/2022, 9:22 PM

Ah actually, I spoke slightly too soon here. I had previously had

"name": "flow"

in my terraform task definition, and then prefect created a second container and

prefect

. Change the container name to prefect resolved that issue. That said, it's still overwriting the custom image I defined and using

prefecthq/prefect:2.6.4-python3.10

instead

Claire Herdeman

11/15/2022, 9:37 PM

Yeah it's registering an additional task definition with everything the same as my definition except a different image.

Claire Herdeman

11/15/2022, 9:44 PM

The

image

param in

ECSTask

seems like a likely culprit. However the value is defaulting to a base image even though the image is already defined in the container definition above

Copy code

"The image to use for the Prefect container in the task. If this value is "
            "not null, it will override the value in the task definition. This value "
            "defaults to a Prefect base image matching your local versions."

Mason Menges

11/15/2022, 10:31 PM

Hey Claire I think you hit the nail on it's head, you have to specify the image as none in the ECStask otherwise it will overwrite the image with the base image, we default to the base image so users don't have to worry about configuring one by default but if you have one defined in your ECS Task Definition setting

image=None

will prevent it from doing so.

👍 1

Mason Menges

11/15/2022, 10:31 PM

Here's the relevant docs https://prefecthq.github.io/prefect-aws/ecs/#prefect_aws.ecs.ECSTask.image

Claire Herdeman

11/16/2022, 3:20 PM

Gochya, yes updating to

image=None

fixed this one! My one nitpick on the wording of that documentation, right now the behavior is: • Specify image to overwrite the image in the task definition • If you don't specify an image value, it will ALSO overwrite your task definition with a default • Explicitly set image=None to use an image that you've specified Behavior in point 2 is a bit confusing, I think it would be helpful to explicitly state point 3

Mason Menges

11/16/2022, 4:18 PM

For sure, We're working on some doc updates I'll make sure this gets passed along as well 😄

🙌 1

Vishnu Duggirala

11/18/2022, 4:55 PM

Anyone's assistance would be greatly appreciated, I have set

image=None

but I get this error

16:14:34.470 | ERROR   | prefect.agent - Failed to submit flow run 'ce9f7889-713c-4d53-96d7-56b70190eccd' to infrastructure.

I have no errors when I explicitly state the image.

Claire Herdeman

11/18/2022, 5:06 PM

I had a few of these issues! There was an issue (that may now be resolved) where prefect 2.7.7 wasn't printing the full traceback for that error, you might check if your agent is running 2.7.7 and if so downgrade to 2.7.6 Also do you have the prefect-aws package installed on the agent?

Vishnu Duggirala

11/18/2022, 5:57 PM

Thanks, let me try that and yes I do have a prefect-aws package installed on my agent

Vishnu Duggirala

11/21/2022, 8:25 PM

This is the traceback I got, it is botocore error,

Container.image should not be null or empty

, task-def arn has the image

Claire Herdeman

11/21/2022, 8:40 PM

Do you have code for your definitions of the ecs task referred to with the

task_definition_arn

Vishnu Duggirala

11/21/2022, 8:48 PM

Yes I do

Vishnu Duggirala

11/21/2022, 8:52 PM

I'm getting this error after the flow run retrieves the task definition and attempts to register it, at which point it throws an error stating that the image should not be null or empty.

Claire Herdeman

11/21/2022, 9:08 PM

Ah so you also have

task_definition_arn=task_arn

, what is the definition for what

task_arn

refers to? Here is my example. Is the image specified in your container definition? if so, is the name of that container

prefect

? I initially had

"name": "flow"

and that was actually causing a second container to spin up that wasn't using my settings, setting

"name": "prefect"

was important for me

Vishnu Duggirala

11/21/2022, 10:06 PM

Yes image is specified in the task-def but no the container image name is not

prefect

, let me try setting the name as

prefect

Vishnu Duggirala

11/21/2022, 10:11 PM

Well that fixed the issue, thanks for all your help @Claire Herdeman

👍 1

Claire Herdeman

11/22/2022, 3:09 PM

Glad that worked!

Claire Herdeman

11/22/2022, 3:11 PM

@Mason Menges, is it already specified somewhere in the docs that if you define a container in your own ecs task definition it must be called

prefect

? I only tried that once I saw two containers running but I could have missed it.

Mason Menges

11/22/2022, 5:33 PM

Hey @Claire Herdeman @Vishnu Duggirala I don't see it in the docs, This actually sounds like it could be a bug would you be willing to open a new issue for this here https://github.com/PrefectHQ/prefect-aws/issues

Claire Herdeman

11/22/2022, 5:37 PM

Ah sure! I can do that in the next day or two, unless Vishnu beats me to it.

Mason Menges

11/22/2022, 5:37 PM

Thanks 🙏

Vishnu Duggirala

11/22/2022, 8:43 PM

I have opened the issue.

🙌 2

7 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.