@Kevin Kho I am unable to access SSM parameters and ENV variables in my flow task. I am using

ECSRun

as follows:

ecs_run = ECSRun(
    run_task_kwargs={'cluster': 'dev-prefect-cluster'},
    task_role_arn='arn:aws:iam::<>:role/dev-task-runner-ecs-task-role',
    execution_role_arn='arn:aws:iam::<>:role/dev-task-runner-ecs-task-execution-role',
    image='<>.<http://dkr.ecr.us-west-1.amazonaws.com/dev-automation-scripts-ecr:latest|dkr.ecr.us-west-1.amazonaws.com/dev-automation-scripts-ecr:latest>',
    labels=['ecs', 'dev']
)
flow.run_config = ecs_run
...

And then in my terraform config for the

prefect-agent

, I am setting a

task_definition

. And this

task_definition_file

is stored in s3 and contains a list of

env

parameters and

secrets

that are pulled from SSM parameter store. What’s happening is that when the flow-task is created, it is not getting any of the

env

and

secrets

that I supply in the task_definition_file. I can kinda see why this is happening: I am not supplying any task_definition in the

ECSRun

config during flow registration. The reason is 2-fold: I don’t want to hard-code these env/SSM parameters in python code, they are supposed to live in terraform configs. And also, we want to share these env and ssm parameters across many tasks, so don’t want to have to define them everytime. Is there a way to accomplish this?

You don’t need to specify it in the ECSRun, but you can also point to a task_definition file in the ECSRun rather than give the task_definition itself. Looking at this, it should work because the agent loads that in and merges it with the run config. What is your error? Also, the roles are the ones that give permission to Parameter store right? So as long as your roles have access, you’ll be able to access it

Here’s the terraform config (copied from above to keep the main channel clean):

container_definitions = templatefile("${path.root}/prefect-agent.json.tftpl",
    {
      region               = data.aws_region.current.name
      prefect_api_key      = var.prefect_api_key
      prefect_api_address  = var.prefect_api_address
      execution_role_arn   = something.prefect_ecs_task_execution_role_arn
      task_role_arn        = something.prefect_ecs_task_role_arn
      security_groups      = something.outputs.prefect_sg_id
      log_group            = aws_cloudwatch_log_group.prefect_agent.name
      cluster              = local.cluster_name
      task_definition_file = "s3://${aws_s3_bucket.prefect_ecs_config.id}/${aws_s3_object.task_definition.id}"
      network_config_file  = "s3://${aws_s3_bucket.prefect_ecs_config.id}/${aws_s3_object.network_config.id}"
    }
  )

Are the env vars secret as well?

no, those are not secrets.

I think you are expecting all env vars on the agent to be carried to the flow automatically? That is not the case. Only env vars defined with:

prefect agent ecs start --env KEY=VALUE

are passed. I think the env vars are being defined in the agent container but not carrying over. You can also define them on

ECSRun(.., env={})

I am not supplying env vars to the prefect agent. btw, specifying

task_definition_path

in

ECSRun

worked. So, now I am supplying task_definition to both the agent and the ECSRun. Prior to doing this, I was only setting

task_definition

on the agent, and the result was that it was not being used/merged for the actual task container.

Ah crud. The merging logic is quite convoluted so I am not surprised if something is wrong. Are you ok defining it in the ECSRun instead?

Correction: I am only supplying

task_definition_path

, not the actual task definition in

ECSRun

. Not ideal but not too bad either since I can still generate the task_definition file in

terraform

and copy it to s3 bucket. So there’s a single source of truth for the task definition.

Ah yes I think it’s only YAML