https://prefect.io logo
Title
c

Claire Herdeman

11/15/2022, 9:11 PM
Hey! A question about using a
task_definition_arn
in an
ECSTask
block, details below
āœ… 1
Here's my current task definition as defined in terraform:
resource "aws_ecs_task_definition" "prefect2Flow" {
  family = "prefect2Flow_${var.env}"
  requires_compatibilities =  ["FARGATE"]
  network_mode = "awsvpc"
  cpu = "512"
  memory = "1024"
  task_role_arn = aws_iam_role.prefect_flow_ecs_task_role.arn
  execution_role_arn = aws_iam_role.prefect_flow_ecs_execution_role.arn
  container_definitions = <<TASK_DEFINITION
  [
    {
        "name": "prefect",
        "image": "${aws_ecr_repository.integrations.repository_url}:latest",
        "environment": [{"name": "PREFECT_DEBUG_MODE", "value": "1"}],
        "logConfiguration": {
            "logDriver": "awslogs",
            "options": {
                "awslogs-group": "${aws_cloudwatch_log_group.prefect_flow.name}",
                "awslogs-region": "us-east-2",
                "awslogs-stream-prefix": "prefect_flow"
            }
        }
    }
  ]
  TASK_DEFINITION
}
And the ECSTask:
ecs = ECSTask(
        task_definition_arn=task,
        cluster=cluster,
        network_mode="awsvpc",
        task_customizations=[
            {
                "op": "replace",
                "path": f"{NETWORK_PATH}/assignPublicIp",
                "value": "DISABLED",
            },
            {
                "op": "add",
                "path": f"{NETWORK_PATH}/subnets",
                "value": get_networking(env, "Subnets"),
            },
            {
                "op": "add",
                "path": f"{NETWORK_PATH}/securityGroups",
                "value": get_networking(env, "SecurityGroups"),
            }
        ]
    )
    ecs.save(f"{env}", overwrite=True)
Ah actually, I spoke slightly too soon here. I had previously had
"name": "flow"
in my terraform task definition, and then prefect created a second container and
prefect
. Change the container name to prefect resolved that issue. That said, it's still overwriting the custom image I defined and using
prefecthq/prefect:2.6.4-python3.10
instead
Yeah it's registering an additional task definition with everything the same as my definition except a different image.
The
image
param in
ECSTask
seems like a likely culprit. However the value is defaulting to a base image even though the image is already defined in the container definition above
"The image to use for the Prefect container in the task. If this value is "
            "not null, it will override the value in the task definition. This value "
            "defaults to a Prefect base image matching your local versions."
m

Mason Menges

11/15/2022, 10:31 PM
Hey Claire I think you hit the nail on it's head, you have to specify the image as none in the ECStask otherwise it will overwrite the image with the base image, we default to the base image so users don't have to worry about configuring one by default but if you have one defined in your ECS Task Definition setting
image=None
will prevent it from doing so.
šŸ‘ 1
c

Claire Herdeman

11/16/2022, 3:20 PM
Gochya, yes updating to
image=None
fixed this one! My one nitpick on the wording of that documentation, right now the behavior is: ā€¢ Specify image to overwrite the image in the task definition ā€¢ If you don't specify an image value, it will ALSO overwrite your task definition with a default ā€¢ Explicitly set image=None to use an image that you've specified Behavior in point 2 is a bit confusing, I think it would be helpful to explicitly state point 3
m

Mason Menges

11/16/2022, 4:18 PM
For sure, We're working on some doc updates I'll make sure this gets passed along as well šŸ˜„
šŸ™Œ 1
v

Vishnu Duggirala

11/18/2022, 4:55 PM
Anyone's assistance would be greatly appreciated, I have set
image=None
but I get this error
16:14:34.470 | ERROR   | prefect.agent - Failed to submit flow run 'ce9f7889-713c-4d53-96d7-56b70190eccd' to infrastructure.
I have no errors when I explicitly state the image.
c

Claire Herdeman

11/18/2022, 5:06 PM
I had a few of these issues! There was an issue (that may now be resolved) where prefect 2.7.7 wasn't printing the full traceback for that error, you might check if your agent is running 2.7.7 and if so downgrade to 2.7.6 Also do you have the prefect-aws package installed on the agent?
v

Vishnu Duggirala

11/18/2022, 5:57 PM
Thanks, let me try that and yes I do have a prefect-aws package installed on my agent
This is the traceback I got, it is botocore error,
Container.image should not be null or empty
, task-def arn has the image
c

Claire Herdeman

11/21/2022, 8:40 PM
Do you have code for your definitions of the ecs task referred to with the
task_definition_arn
?
v

Vishnu Duggirala

11/21/2022, 8:48 PM
Yes I do
I'm getting this error after the flow run retrieves the task definition and attempts to register it, at which point it throws an error stating that the image should not be null or empty.
c

Claire Herdeman

11/21/2022, 9:08 PM
Ah so you also have
task_definition_arn=task_arn
, what is the definition for what
task_arn
refers to? Here is my example. Is the image specified in your container definition? if so, is the name of that container
prefect
? I initially had
"name": "flow"
and that was actually causing a second container to spin up that wasn't using my settings, setting
"name": "prefect"
was important for me
v

Vishnu Duggirala

11/21/2022, 10:06 PM
Yes image is specified in the task-def but no the container image name is not
prefect
, let me try setting the name as
prefect
Well that fixed the issue, thanks for all your help @Claire Herdeman
šŸ‘ 1
c

Claire Herdeman

11/22/2022, 3:09 PM
Glad that worked!
@Mason Menges, is it already specified somewhere in the docs that if you define a container in your own ecs task definition it must be called
prefect
? I only tried that once I saw two containers running but I could have missed it.
m

Mason Menges

11/22/2022, 5:33 PM
Hey @Claire Herdeman @Vishnu Duggirala I don't see it in the docs, This actually sounds like it could be a bug would you be willing to open a new issue for this here https://github.com/PrefectHQ/prefect-aws/issues
c

Claire Herdeman

11/22/2022, 5:37 PM
Ah sure! I can do that in the next day or two, unless Vishnu beats me to it.
m

Mason Menges

11/22/2022, 5:37 PM
Thanks šŸ™
v

Vishnu Duggirala

11/22/2022, 8:43 PM
I have opened the issue.
šŸ™Œ 2