Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

I'm testing out the ECS workpool/worker and deployments. I'm curious, is there a convenient way to inject secrets from either secrets manager or parameter store? I'm trying to not use blocks because I'd rather not store my AWS credentials in prefect cloud. Ideally there would be a job_variables option that would allow me to set secrets in the container_definition.

Screenshot 2023-08-02 at 4.35.56 PM.png

hey, I believe you can do this from the advanced tab on the work pool

you can add whatever parameters you'd like to `containerDefinitions` and they'll get passed to `registerTaskDefinition`

the ones shown are just the common ones we surface

ok, let me play around with that. Thanks, Jake.

<@U02EBC3N05N> - following up on this, I've gotten it to work by manually editing the Advance tab using the UI. However, ideally this would be something that could be configured via deployments in prefect.yaml. Is that currently possible? It doesn't seem like it is. I tried adding the containerDefinitions to the `job_variables` but it does not get added to the task definition.

The reason this is important for my use case is that I have different teams at my company using Prefect. Each team has different needs for networking, secrets, volumes, etc. Ideally at a flow level, I could be able to control these settings. By hard coding it into the Advance tab, the task definition becomes very static.

Ideally, we'd like to be able to set the following items per flow deployment using job_variables.

• ephemeralStorage
• secrets (in containerDefinitions)
• subnets/security groups (awsvpcConfiguration)
I see that <https://github.com/PrefectHQ/prefect-aws/issues/298|this issue> covers my third point. I could look at making a PR to add these things. Is Prefect open to having these configurations added to the base template?

Hey! All of that should be possible already:
<https://docs.prefect.io/2.11.3/concepts/work-pools/#base-job-template>

While you can hard code things into the work pool template (like you've done), you're right that  it will be the same for every deployment.

Under the advanced section you can add temptable job_variables, which are then able to be set on every deployment differently. You'll also need to add the temptable value into the job_configuration (like `"my_extra_variable": {{ job_variable }}`

You can also edit the base template to include values that the ECS API accepts, even if they're not already in the base template