https://prefect.io
Join Slack
Hi everyone, We are trying to run our prefect flows in an ECS cluster and we need to run different f...
# ask-community
s
Hi everyone, We are trying to run our prefect flows in an ECS cluster and we need to run different flows with different secrets and different container configs is there a way we can achieve this using prefect.yaml
j
you should be able to define the specific configuration you need either in the deployment or as an override to the workpool
s
We tried doing that but when we see our ecs task definition the configuration does not replicate there
this what our deployment looks like
So the secrets section and the ephemeral storage does not replicate to the ecs task that is created using this deployment
the work around we found for this was to edit the advanced section of the work pool manually and add these things under job_configuration and save it.
j
ah yes exactly!
s
Just to confirm that it can be done manually only we can’t use prefect.yaml file for this?
j
yes you can! you can update your work pool job variables (at the deployment level) with the specified config you want to override. You’ll also need to update your workpool for ECS under the advanced tab to expose the variables for the secret // ephemeral storage (under 
variables
) and then update the 
job_configuration
to reference the variable you created using jinga
so something like this:
Copy code
{
  "variables": {
    "type": "integer",
    "title": "sizeInGiB",
    "description": "description"
   ...
  },
  "job_configuration": {
    "task_definition": {
      "ephemeralStorage": {
        "sizeInGiB": "{{ sizeInGiB }}"
      }
    }
  }
}
(abbreviated)
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by