https://prefect.io logo
Join Slack
Powered by
Hi everyone — I’ve got a security concern I’m not ...
# ask-community
b
Hi everyone — I’ve got a security concern I’m not sure how to solve. When running a flow as an ECS Task with a custom task definition, prefect injects a bunch of environment variables which then appear on the ECS Task execution screen on AWS console. This includes the prefect api keys. Does anyone know how to solve that?
a
Isn't it just a list, and so someone with AWS access to view won't be able to read the value? I'm thinking of how I use secure strings in parameter store for things like API keys and then create task definitions with corresponding environment variables, but there's no security issue inherent in that as far as I know: developers from other teams can see the list of environment variables in the task definition without having access to the unencrypted value in the parameter store. There might be a security issue if the flow itself accesses these API keys and leaks them somewhere, but that should be auditable, right?
upvote 1
b
@Amanda Wee is not a list!
a
Ahhh... okay, yeah, that's not good.
b
the problem is that the actual task definition is fine and doesn’t have those vars, but Prefect uses the “containerOverrides” to inject them
so they only appear on the ecs task execution
a
@Bruno Murino I was just about to answer in the previous thread, but it’s actually better to open a separate thread for that, so thank you! I would be happy to review your PR if you would want to contribute. But I also believe that if you follow the tutorial I shared that uses AWS Parameter Store ARN, you should not have the API Key stored on the task definition in plain text, but rather as ARN pointing to the Parameter in the AWS Parameter Store. I believe something must have gone wrong in the task definition you use for the flow, because your agent’s task definition doesn’t have that problem. If you would share your flow’s ECSRun incl. your output of 
prefect diagnostics
, I would reproduce and investigate the issue in more depth.
b
Hi Anna! This is the ECSRun — let me get the prefect diagnostics (not familiar with it)
a
it’s just a prefect CLI command
b
this section of the github code you shared yesterday, in my view, is the cause of the problem https://github.com/PrefectHQ/prefect/blob/d44b72a950ebda9f7bc6a9712fc71e2e9c680d25/src/prefect/agent/ecs/agent.py#L491-L526
a
Thanks for sharing! I think I see the problem. You don’t pass the secret in your containerDefinitions, that’s why it’s taken from the environment variable on the machine from which you register your flows. Adding this to the 
containerDefinitions
list in your ECSRun should solve the problem:
Copy code
"secrets": [
        {
          "name": "PREFECT__CLOUD__API_KEY",
          "valueFrom": "arn:aws:ssm:us-east-1:YOUR_ACCOUNT_ID:parameter/YOUR_PARAMETER_NAME"
        }
      ]
upvote 1
Here is a full task definition example that you could use with EC2 data plane:
Copy code
{
  "family": "prefectFlow",
  "requiresCompatibilities": [
    "EC2"
  ],
  "networkMode": "awsvpc",
  "cpu": "512",
  "memory": "1024",
  "taskRoleArn": "arn:aws:iam::123456789:role/prefectTaskRole",
  "executionRoleArn": "arn:aws:iam::123456789:role/prefectECSAgentTaskExecutionRole",
  "containerDefinitions": [
    {
      "name": "flow",
      "image": "<http://123456789.dkr.ecr.us-east-1.amazonaws.com/community:latest|123456789.dkr.ecr.us-east-1.amazonaws.com/community:latest>",
      "essential": true,
      "environment": [
        {
          "name": "AWS_RETRY_MODE",
          "value": "adaptive"
        },
        {
          "name": "AWS_MAX_ATTEMPTS",
          "value": "10"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/prefectEcsAgent",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs",
          "awslogs-create-group": "true"
        }
      },
      "secrets": [
        {
          "name": "PREFECT__CLOUD__API_KEY",
          "valueFrom": "arn:aws:ssm:us-east-1:123456789:parameter/PREFECT__CLOUD__API_KEY"
        }
      ]
    }
  ]
}
b
nice! I’m trying your suggestion now
@Anna Geller didn’t work 😞 this is the registered task definition
and this is the task execution of it
a
@Bruno Murino ok I see what you mean. The task execution in the UI indeed shows the env variables that have been used during the execution of this specific task, incl. those retrieved from secrets. But I’m not sure if this is something we can address on the Prefect side - if you wanna try, I’d be happy to review your PR. But my feeling is that you would need to restrict this via AWS console permissions perhaps? In general, this information is not stored anywhere persistent and the task execution view in the console is only available for a very short time winow (hard to say but maybe an hour or max 3?).
b
I think it can be address but accepting additional arguments on https://github.com/PrefectHQ/prefect/blob/d44b72a950ebda9f7bc6a9712fc71e2e9c680d25/src/prefect/agent/ecs/agent.py#L491-L526 I see no reason why an option for the “api token” to be fetched from the parameter/secret store wouldn’t be possible
I’m happy to try that! I’ll get informed on how to contribute and etc
a
api_tokens are deprecated and shouldn’t be used going forward
b
ah sorry I didn’t mean token exactly
I meant the api key
a
b
thanks!
and thanks Anna for all the help! You’ve been super helpful!
🙌 1
2 Views
Open in Slack
PreviousNext
Powered by