https://prefect.io logo
m

Mark McDonald

08/02/2023, 8:31 PM
I'm testing out the ECS workpool/worker and deployments. I'm curious, is there a convenient way to inject secrets from either secrets manager or parameter store? I'm trying to not use blocks because I'd rather not store my AWS credentials in prefect cloud. Ideally there would be a job_variables option that would allow me to set secrets in the container_definition.
āœ… 1
j

Jake Kaplan

08/02/2023, 8:36 PM
hey, I believe you can do this from the advanced tab on the work pool
you can add whatever parameters you'd like to
containerDefinitions
and they'll get passed to
registerTaskDefinition
the ones shown are just the common ones we surface
m

Mark McDonald

08/02/2023, 8:38 PM
ok, let me play around with that. Thanks, Jake.
šŸ‘ 1
@Jake Kaplan - following up on this, I've gotten it to work by manually editing the Advance tab using the UI. However, ideally this would be something that could be configured via deployments in prefect.yaml. Is that currently possible? It doesn't seem like it is. I tried adding the containerDefinitions to the
job_variables
but it does not get added to the task definition. The reason this is important for my use case is that I have different teams at my company using Prefect. Each team has different needs for networking, secrets, volumes, etc. Ideally at a flow level, I could be able to control these settings. By hard coding it into the Advance tab, the task definition becomes very static.
Ideally, we'd like to be able to set the following items per flow deployment using job_variables. ā€¢ ephemeralStorage ā€¢ secrets (in containerDefinitions) ā€¢ subnets/security groups (awsvpcConfiguration) I see that this issue covers my third point. I could look at making a PR to add these things. Is Prefect open to having these configurations added to the base template?
j

Jake Kaplan

08/09/2023, 8:43 PM
Hey! All of that should be possible already: https://docs.prefect.io/2.11.3/concepts/work-pools/#base-job-template While you can hard code things into the work pool template (like you've done), you're right that it will be the same for every deployment. Under the advanced section you can add temptable job_variables, which are then able to be set on every deployment differently. You'll also need to add the temptable value into the job_configuration (like
"my_extra_variable": {{ job_variable }}
You can also edit the base template to include values that the ECS API accepts, even if they're not already in the base template
m

Mark McDonald

08/10/2023, 8:02 PM
this worked - it's amazing! thank you
šŸ™Œ 1