Daniel Rodriguez Ciotti
11/12/2025, 10:45 AMMarvin
11/12/2025, 10:45 AMMarvin
11/12/2025, 10:55 AMrun_deployment(...) and set job_variables={"cpu": ..., "memory": ...} along with your parameters.
2) From CLI: use prefect deployment run ... -jv cpu=... -jv memory=... at trigger time.
Examples
- Python wrapper that routes resources by input size and triggers the run:
from prefect.deployments import run_deployment
def size_to_resources(n: int) -> tuple[int, int]:
# Example Fargate-friendly combos; adjust to your needs
# See AWS Fargate CPU/memory combos
if n < 100:
return 256, 512 # 0.25 vCPU, 0.5 GB
elif n < 1000:
return 1024, 2048 # 1 vCPU, 2 GB
else:
return 2048, 4096 # 2 vCPU, 4 GB
async def trigger(items: list[dict]):
cpu, memory = size_to_resources(len(items))
await run_deployment(
name="my-flow/my-deployment",
parameters={"items": items},
job_variables={"cpu": cpu, "memory": memory},
)
- CLI (values are parsed as JSON)
prefect deployment run "my-flow/my-deployment" \
-p items='[{"a":1},{"a":2}]' \
-jv cpu=1024 -jv memory=2048
Notes and constraints
- In Prefect 3.x, job variables can be overridden per run at trigger time (API/CLI), but cannot be changed by code inside the same run.
- ECS Fargate enforces valid CPU/memory combinations. Make sure your function returns a valid pair. See AWS docs for allowed combinations: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html
- The job variable keys for ECS CPU and memory are typically cpu and memory on the task definition; you can confirm in your work pool’s base job template. Guidance on customizing job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables
- If you need this logic to happen “inside Prefect,” a common pattern is a lightweight “router” flow that computes the resources, then triggers a second flow run of the same deployment with job_variables set appropriately (the router run remains small, the child run gets the larger resources).
- Jinja/expression templating of job_variables based on flow parameters is not supported.
Would you like an example tailored to how you trigger runs today (UI, CLI, or Python), and what resource tiers you want to use? Relevant docs:
- Customizing job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables
- ECS guide: https://docs-3.prefect.io/integrations/prefect-aws/ecs_guide