Does anyone know how to override the

ephemeralStorage

parameter for ECS Tasks in Prefect Cloud? I tried adding the snippet below to the Job Variables configuration, but based on the logs it looks like each ECS Task is still using the default (20 GiB). Using this configuration works to override the default

cpu

and

memory

parameters, so it’s unclear why overriding this parameter isn’t working:

"ephemeralStorage": {
    "sizeInGiB": 40
  }

Hi Bryce, are you sure there is an ephemeralStorage parameter in the base-job-template of your work pool? Are you using a ecs or ecs:push work pool? You can get the template with this command (or in the advanced tab when you edit your work pool in the ui): prefect work-pool get-default-base-job-template --type ecs:push

Thanks Mira! I’m using an ecs work pool (not push). The base template does not appear to have an

ephemeralStorage

parameter defined. Are you saying that I need to define a default one here before I can override it in the job variables configuration for a specific flow? Or are you saying that overriding this parameter is simply not supported by Prefect?

Well, I didn't have the need to set parameters that are not in the base template, yet, but you may want to have a look to this video: ( minutes: 33:45), there they adjust the job template as easy as with adjusting the underlaying json (advanced tab of the work pool), to ingest the needed parameters (note: you also have to set the parameter down at the bottom for the job configuration)

Thanks Mina! Adding to the base job template worked. Thanks for the guidance! Still trying to figure out flow-specific overrides, but this is what I added to my base job template of the workpool in case it’s useful for other folks for future reference: In variables-->properties:

"ephemeralStorage": {
        "type": "object",
        "title": "Ephemeral Storage",
        "description": "The storage available on the task device.",
        "additionalProperties": {
          "sizeInGiB": "int"
        }
      }

In job_configuration:

"ephemeralStorage": "{{ ephemeralStorage }}",

Then finally in the job variable configurations in the Prefect Flow:

"ephemeralStorage": {
        "sizeInGiB": 40 # or whatever size is needed up to 200
      },

Did you link the parameter to the job configuration in jinja notation like in the screenshot? I think this is necessary to assign it to the correct aws task definition parameters defined here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-definition-template.html Since ephemeralStorage is on the same level as memory and cpu in my opinion (i am not an expert) it should work . I do my deployment specific overrides like this and it works (I am using an ecs:push work pool):

job_variables = {
    "execution_role_arn": os.getenv("EXECUTION_ROLE", ""),
    "task_role_arn": os.getenv("TASK_ROLE", ""),
    "cluster": os.getenv("ECS_CLUSTER"),
    "vpc_id": os.getenv("VPC_ID", ""),
    "container_name": os.getenv("ECR_REPO_NAME", ""),
    "family": "prefect-flow",
    "aws_credentials": {
        "$ref": {
             "block_document_id": os.getenv("AWS_CREDENTIAL_BLOCK_ID", "")
         }
     },
}

Ah that resolved it! Updated my comment above to reflect the correct configuration requirements. Thanks again! 🙏

Old conversation, but i needed to increased the size of ephemeral storage for a flow. I load ECS configuration from a block variable, and simply updating the ECS task defintion on my blocks to include this on the Task Defintion paramter, did the trick

{
  "ephemeralStorage": {
    "sizeInGiB": 40
  }
}