https://prefect.io logo
s

Sagar Arora

08/09/2023, 2:15 PM
Hi everyone, We are trying to run our prefect flows in an ECS cluster and we need to run different flows with different secrets and different container configs is there a way we can achieve this using prefect.yaml
j

Jamie Zieziula

08/09/2023, 3:55 PM
you should be able to define the specific configuration you need either in the deployment or as an override to the workpool
s

Sagar Arora

08/09/2023, 3:56 PM
We tried doing that but when we see our ecs task definition the configuration does not replicate there
this what our deployment looks like
So the secrets section and the ephemeral storage does not replicate to the ecs task that is created using this deployment
the work around we found for this was to edit the advanced section of the work pool manually and add these things under job_configuration and save it.
j

Jamie Zieziula

08/09/2023, 5:06 PM
ah yes exactly!
s

Sagar Arora

08/09/2023, 5:07 PM
Just to confirm that it can be done manually only we can’t use prefect.yaml file for this?
j

Jamie Zieziula

08/09/2023, 10:59 PM
yes you can! you can update your work pool job variables (at the deployment level) with the specified config you want to override. You’ll also need to update your workpool for ECS under the advanced tab to expose the variables for the secret // ephemeral storage (under
variables
) and then update the
job_configuration
to reference the variable you created using jinga
so something like this:
Copy code
{
  "variables": {
    "type": "integer",
    "title": "sizeInGiB",
    "description": "description"
   ...
  },
  "job_configuration": {
    "task_definition": {
      "ephemeralStorage": {
        "sizeInGiB": "{{ sizeInGiB }}"
      }
    }
  }
}
(abbreviated)