Channels
#prefect-community
Title
s
Sagar Arora
08/09/2023, 2:15 PMHi everyone, We are trying to run our prefect flows in an ECS cluster and we need to run different flows with different secrets and different container configs is there a way we can achieve this using prefect.yaml
j
Jamie Zieziula
08/09/2023, 3:55 PMyou should be able to define the specific configuration you need either in the deployment or as an override to the workpool
s
Sagar Arora
08/09/2023, 3:56 PMWe tried doing that but when we see our ecs task definition the configuration does not replicate there
this what our deployment looks like
So the secrets section and the ephemeral storage does not replicate to the ecs task that is created using this deployment
the work around we found for this was to edit the advanced section of the work pool manually and add these things under job_configuration and save it.
j
Jamie Zieziula
08/09/2023, 5:06 PMah yes exactly!
s
Sagar Arora
08/09/2023, 5:07 PMJust to confirm that it can be done manually only we can’t use prefect.yaml file for this?
j
Jamie Zieziula
08/09/2023, 10:59 PMyes you can! you can update your work pool job variables (at the deployment level) with the specified config you want to override. You’ll also need to update your workpool for ECS under the advanced tab to expose the variables for the secret // ephemeral storage (under
variablesjob_configurationso something like this:
Copy code
{
"variables": {
"type": "integer",
"title": "sizeInGiB",
"description": "description"
...
},
"job_configuration": {
"task_definition": {
"ephemeralStorage": {
"sizeInGiB": "{{ sizeInGiB }}"
}
}
}
}(abbreviated)