https://prefect.io logo
Channels
Powered by
c

Claire Herdeman

11/15/2022, 9:11 PM
Hey! A question about using a 
task_definition_arn
in an 
ECSTask
block, details below
āœ… 1
Here's my current task definition as defined in terraform:
Copy code
resource "aws_ecs_task_definition" "prefect2Flow" {
  family = "prefect2Flow_${var.env}"
  requires_compatibilities =  ["FARGATE"]
  network_mode = "awsvpc"
  cpu = "512"
  memory = "1024"
  task_role_arn = aws_iam_role.prefect_flow_ecs_task_role.arn
  execution_role_arn = aws_iam_role.prefect_flow_ecs_execution_role.arn
  container_definitions = <<TASK_DEFINITION
  [
    {
        "name": "prefect",
        "image": "${aws_ecr_repository.integrations.repository_url}:latest",
        "environment": [{"name": "PREFECT_DEBUG_MODE", "value": "1"}],
        "logConfiguration": {
            "logDriver": "awslogs",
            "options": {
                "awslogs-group": "${aws_cloudwatch_log_group.prefect_flow.name}",
                "awslogs-region": "us-east-2",
                "awslogs-stream-prefix": "prefect_flow"
            }
        }
    }
  ]
  TASK_DEFINITION
}
And the ECSTask:
Copy code
ecs = ECSTask(
        task_definition_arn=task,
        cluster=cluster,
        network_mode="awsvpc",
        task_customizations=[
            {
                "op": "replace",
                "path": f"{NETWORK_PATH}/assignPublicIp",
                "value": "DISABLED",
            },
            {
                "op": "add",
                "path": f"{NETWORK_PATH}/subnets",
                "value": get_networking(env, "Subnets"),
            },
            {
                "op": "add",
                "path": f"{NETWORK_PATH}/securityGroups",
                "value": get_networking(env, "SecurityGroups"),
            }
        ]
    )
    ecs.save(f"{env}", overwrite=True)
Ah actually, I spoke slightly too soon here. I had previously had 
"name": "flow"
in my terraform task definition, and then prefect created a second container and 
prefect
. Change the container name to prefect resolved that issue. That said, it's still overwriting the custom image I defined and using 
prefecthq/prefect:2.6.4-python3.10
instead
Yeah it's registering an additional task definition with everything the same as my definition except a different image.
The 
image
param in 
ECSTask
seems like a likely culprit. However the value is defaulting to a base image even though the image is already defined in the container definition above
Copy code
"The image to use for the Prefect container in the task. If this value is "
            "not null, it will override the value in the task definition. This value "
            "defaults to a Prefect base image matching your local versions."
m

Mason Menges

11/15/2022, 10:31 PM
Hey Claire I think you hit the nail on it's head, you have to specify the image as none in the ECStask otherwise it will overwrite the image with the base image, we default to the base image so users don't have to worry about configuring one by default but if you have one defined in your ECS Task Definition setting 
image=None
will prevent it from doing so.
šŸ‘ 1
c

Claire Herdeman

11/16/2022, 3:20 PM
Gochya, yes updating to 
image=None
fixed this one! My one nitpick on the wording of that documentation, right now the behavior is: ā€¢ Specify image to overwrite the image in the task definition ā€¢ If you don't specify an image value, it will ALSO overwrite your task definition with a default ā€¢ Explicitly set image=None to use an image that you've specified Behavior in point 2 is a bit confusing, I think it would be helpful to explicitly state point 3
m

Mason Menges

11/16/2022, 4:18 PM
For sure, We're working on some doc updates I'll make sure this gets passed along as well šŸ˜„
šŸ™Œ 1
v

Vishnu Duggirala

11/18/2022, 4:55 PM
Anyone's assistance would be greatly appreciated, I have set 
image=None
but I get this error 
16:14:34.470 | ERROR   | prefect.agent - Failed to submit flow run 'ce9f7889-713c-4d53-96d7-56b70190eccd' to infrastructure.
I have no errors when I explicitly state the image.
c

Claire Herdeman

11/18/2022, 5:06 PM
I had a few of these issues! There was an issue (that may now be resolved) where prefect 2.7.7 wasn't printing the full traceback for that error, you might check if your agent is running 2.7.7 and if so downgrade to 2.7.6 Also do you have the prefect-aws package installed on the agent?
v

Vishnu Duggirala

11/18/2022, 5:57 PM
Thanks, let me try that and yes I do have a prefect-aws package installed on my agent
This is the traceback I got, it is botocore error, 
Container.image should not be null or empty
, task-def arn has the image
c

Claire Herdeman

11/21/2022, 8:40 PM
Do you have code for your definitions of the ecs task referred to with the 
task_definition_arn
?
v

Vishnu Duggirala

11/21/2022, 8:48 PM
Yes I do
I'm getting this error after the flow run retrieves the task definition and attempts to register it, at which point it throws an error stating that the image should not be null or empty.
c

Claire Herdeman

11/21/2022, 9:08 PM
Ah so you also have 
task_definition_arn=task_arn
, what is the definition for what 
task_arn
refers to? Here is my example. Is the image specified in your container definition? if so, is the name of that container 
prefect
? I initially had 
"name": "flow"
and that was actually causing a second container to spin up that wasn't using my settings, setting 
"name": "prefect"
was important for me
v

Vishnu Duggirala

11/21/2022, 10:06 PM
Yes image is specified in the task-def but no the container image name is not 
prefect
, let me try setting the name as 
prefect
Well that fixed the issue, thanks for all your help @Claire Herdeman
šŸ‘ 1
c

Claire Herdeman

11/22/2022, 3:09 PM
Glad that worked!
@Mason Menges, is it already specified somewhere in the docs that if you define a container in your own ecs task definition it must be called 
prefect
? I only tried that once I saw two containers running but I could have missed it.
m

Mason Menges

11/22/2022, 5:33 PM
Hey @Claire Herdeman @Vishnu Duggirala I don't see it in the docs, This actually sounds like it could be a bug would you be willing to open a new issue for this here https://github.com/PrefectHQ/prefect-aws/issues
c

Claire Herdeman

11/22/2022, 5:37 PM
Ah sure! I can do that in the next day or two, unless Vishnu beats me to it.
m

Mason Menges

11/22/2022, 5:37 PM
Thanks šŸ™
v

Vishnu Duggirala

11/22/2022, 8:43 PM
I have opened the issue.
šŸ™Œ 2
7 Views
Powered by