Howard Cornwell
01/17/2024, 2:52 PMcpu/memory
allocations on a per-flow basis.
Right now, the implementation includes
• An ECS cluster with an ECS worker running on it
• A single Docker image containing all code for all flows
• A base job template on the ECS work pool that specifies the cluster and image
The ECS worker spawns flow runs on the cluster, but it always spawns flow runs in containers with the default 1024/2048
cpu/memory
allocations
I've been trying to override these per-flow using Deployment.build_from_flow()
. See the attached example.
The commented out sections are the ways I've tried overriding the cpu/memory
allocations, but none have worked.
I'd prefer to not have to build each individual flow into separate docker images. We've got close to 100 flows that all run out of the same codebase, so having a single image seems the sensible way to do things.
I feel like I'm missing something crucial. Any pointers would be much appreciated.Nate
01/17/2024, 3:40 PMinfra_overrides
should be what you want here, you're "overriding" the value on a work pool for a specific deployment (without changing the work pool directly)
can you explain whats not working in that regard? what does the Configuration tab show in the UI after building a deployment with an infra_overrides
?Nate
01/17/2024, 3:41 PMECS_DEFAULT_CPU
ECS_DEFAULT_MEMORY
are not fields on the ECS work pool, i think its just cpu
and memory
respectively. so i think you just want
# infra_overrides={,
# "cpu": "512",
# "memory": "1024",
# },
Howard Cornwell
01/17/2024, 3:44 PMNate
01/17/2024, 3:45 PM