For ECS workers in Prefect, you can define AWS Secrets Manager secrets in your

prefect.yaml

file using the following structure:

yaml
deployments:
  - name: your-deployment-name
    work_pool:
      name: your-ecs-pool
      work_queue_name: default
    job:
      variables:
        secrets:
          - name: "USERNAME"  # This will be the environment variable name in the container
            valueFrom: "arn:aws:secretsmanager:region:account:secret:secret-name"
          - name: "PASSWORD"
            valueFrom: "arn:aws:secretsmanager:region:account:secret:another-secret-name"

A few important points: 1. Your ECS task role must have permissions to access the secrets in AWS Secrets Manager 2. The

valueFrom

field should contain the full ARN of your secret 3. The

name

field defines the environment variable name that will be available in your container Make sure your ECS task execution role has the following permissions:

json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue"
            ],
            "Resource": [
                "arn:aws:secretsmanager:region:account:secret:secret-name"
            ]
        }
    ]
}

You can find more details about setting up ECS workers in the Prefect AWS ECS guide. Note: Be cautious when using secrets with ECS workers, as there is a known issue where some secret values might be visible in plain text in certain contexts. If you're handling particularly sensitive information, you may want to implement additional security measures.

that's the ideal outcome, which i've done before using vault + envconsul. im not tied to this solution, but i do want to cut down on the latency for fetching secrets. rn we fetch each one individually and startup times are really slow. the alternative is to nest all secrets into a namespace and fetch that from secrets manager, just one http request (not bad at all). im fine w this approach, but secrets on the container feel a bit more idiomatic w ecs

but ultimately, as stated, if you want the values in your env so the dotenv source for pydantic settings can find it, you'll need to populate

job_variables.env

for a flow run with your vault secrets one way or another

the others seem hacky. if you guys have something ootb for this, then that's better for the next person who has to grok what i'm setting up rn

@Marvin quick rundown on secret blocks

Here's your rundown on Secret blocks in Prefect: 1. Basic Usage:

python
from prefect.blocks.system import Secret

# Create and save a secret
secret_block = Secret(value="your-sensitive-value")
secret_block.save(name="my-secret-name")

# Load and use a secret
secret_value = Secret.load("my-secret-name").get()

2. Key Features: - 🔒 Encrypted Storage: Values are encrypted at rest - 👁️ Masked Display: Secrets are automatically masked in logs and the UI - 🔄 Version Control Safe: Can be used safely with version control as secrets are stored in Prefect's backend - 🎯 Flow Integration: Easily accessible within flows and tasks 3. Common Use Cases:

python
@flow
def my_flow():
    # Load API key from secret block
    api_key = Secret.load("api-key").get()
    
    # Use in API calls or other sensitive operations
    response = make_api_call(api_key=api_key)

4. CLI Management:

bash
# Create a secret via CLI
prefect block register -m prefect.blocks.system

# View available secrets
prefect block ls

5. Best Practices: - Use descriptive names for your secrets - Rotate secrets periodically - Don't log secret values or include them in error messages - Consider using environment variables for local development 6. UI Management: - Secrets can be managed through the Prefect UI under Blocks → Secret - Values can be updated without changing flow code The Secret block is particularly useful when you need to: - Store API keys for service integrations - Manage database credentials - Handle authentication tokens - Store any sensitive configuration that shouldn't be in code Remember that Secret blocks are meant for storing individual secret values. For more complex configurations involving multiple secrets, consider using other specialized blocks or combining multiple Secret blocks as needed. You can find more details in the Prefect documentation on storing secrets.

wish though that we could pass configuration through that aren't defined in the base_job_template

def don't want to manage keeping secrets synced btw a prefect block and our secret store

on making custom steps: • custom step def • usage

to do this i mean https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-tutorial.html#:~:text=%22secrets%22%[…]20%20%20%20%20%20%5D%2C