Dev Dabke
06/13/2024, 4:50 PM"ephemeralStorage": {
"sizeInGiB": 200
}
Kevin Grismore
06/13/2024, 4:56 PMKevin Grismore
06/13/2024, 4:56 PM"ephemeralStorage": {
"sizeInGiB": "{{ ephemeral_storage }}"
}
Kevin Grismore
06/13/2024, 4:57 PMDev Dabke
06/13/2024, 4:58 PMKevin Grismore
06/13/2024, 5:00 PMvariables
object with a name and type so it gets picked up as a job_variable. once you do that it'll show up in the defaults tab for the work pool tooDev Dabke
06/13/2024, 5:02 PMDev Dabke
06/13/2024, 5:47 PM{
"ephemeralStorage": {
"sizeInGiB": "{{ ephemeral_storage }}"
},
"variables": {
"type": "object",
"required": [
"aws_credentials"
],
"properties": {
"ephemeral_storage": {
"type": "integer",
"title": "Ephemeral Storage (GiB)",
"default": 20,
"description": "The amount of ephemeral storage to provide the ECS task (valid amounts are between 20 and 200)."
},
Dev Dabke
06/13/2024, 5:48 PMDev Dabke
06/13/2024, 5:57 PMDev Dabke
06/13/2024, 5:57 PMJames Gatter
09/05/2024, 6:14 PM"variables": {
"properties": {
"ephemeral_storage": {
"type": "integer",
"title": "Ephemeral Storage (GiB)",
"default": 21,
"description": "..."
},
...
},
...
},
I chose 21
for the default since I got this boto3 error logged when I tried the default of 20:
Flow run could not be submitted to infrastructure: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: EphemeralStorage size should be at least 21
01:56:24 PM
prefect_cloud.push_work_pool
The argument gets plugged in under "job_configuration" -> "task_definition" within "ephemeralStorage" -> "sizeInGiB":
"job_configuration": {
...
"task_definition": {
...
"ephemeralStorage": {
"sizeInGiB": "{{ ephemeral_storage }}"
},
...
},
}
James Gatter
09/05/2024, 6:15 PM