Short answer - Yes: in Prefect 3.x you can set ECS CPU/memory per deployment in prefect.yaml via the deployment’s work_pool job_variables. - No: Prefect doesn’t auto-size ECS resources based on “how heavy” your run is. You’ll need to choose a size at (or before) run time. There are a few good patterns below. How to set CPU/memory in prefect.yaml - Define defaults in the ECS work pool’s base job template (networking, roles, etc.) - Override per deployment with job_variables Example:

# prefect.yaml
name: my-project

deployments:
- name: etl-heavy
  entrypoint: flows/etl.py:flow
  work_pool:
    name: my-ecs-pool
    job_variables:
      # ECS CPU units and MiB of memory (must be a valid Fargate combo)
      cpu: 2048
      memory: 4096
      image: <aws_account_id>.<http://dkr.ecr.us-east-1.amazonaws.com/my-image:latest|dkr.ecr.us-east-1.amazonaws.com/my-image:latest>

Notes - cpu/memory must be valid Fargate task sizes (CPU units, memory in MiB). See AWS’s allowed combinations: AWS ECS task size combinations - Most other infrastructure (cluster, subnets, security groups, roles) lives in the work pool’s base job template; use job_variables for per-deployment tweaks like cpu/memory/image. Adapting resources to “flow intensity” - Built-in auto-sizing: not available. Prefect doesn’t inspect your workload and pick a Fargate size for you. - Pragmatic patterns: 1) Multiple deployments for the same flow: small/medium/large, each with different cpu/memory. Trigger the right one based on your own heuristics. 2) Per-run overrides of job variables: when you trigger a run, override cpu/memory in job variables. You can do this via the UI (Run > Advanced/Overrides) or the API. For CLI, check

prefect deployment run --help

for a

--job-variables

option. 3) Template your pool’s job template to read from job variables with defaults, e.g.

cpu: "{{ job_variables.cpu | default(1024) }}"

, then pass cpu/memory only when you need to upsize. 4) Upstream router: have a lightweight “router” flow decide the size (S/M/L) and then call the appropriate deployment via the API with the desired job_variables. What autoscaling can help with - ECS service autoscaling helps with how many worker processes you run (i.e., how quickly you can launch new tasks), not the size of each task. Pair it with Prefect work pool queues to keep throughput high if your queue backs up, but you still decide task size before each run. Docs for reference - Customizing job variables: Customize job variables - ECS worker guide: Prefect AWS ECS guide If you share your current prefect.yaml and what sizes you need (S/M/L), I can sketch the exact templates and a clean promotion path. I know, thrilling.

Great question. With DaskTaskRunner, your CPU/memory settings matter a lot—but which settings matter depends on where your Dask workers actually run. Assuming Prefect 3.x and prefect-dask. Two common setups 1) Local Dask cluster inside your ECS task (the default with DaskTaskRunner) - Your ECS task’s cpu/memory are a hard ceiling for the Dask cluster. - Tune Dask to fit inside that container: - CPU-bound tasks: prefer processes, not threads. Roughly match n_workers to vCPUs and set threads_per_worker=1. - I/O-bound tasks: threads per worker > 1 can help; fewer processes. - Memory-heavy tasks: set memory_limit per worker to leave headroom for the scheduler/process and the Python driver. - If you need “bigger,” you must give the ECS task more CPU/memory. You cannot exceed the container’s limits. 2) External Dask cluster (address="...") on separate infra - Your Prefect/ECS task just orchestrates; heavy work runs on the Dask cluster. - In this case, the ECS task can be small; scale CPU/memory on the Dask workers instead (K8s, ECS, VMs…). - Best for truly expensive flows that need >1 machine. Practical tuning for expensive runs - CPU-bound Python code (GIL): - Use processes (the default for LocalCluster). - n_workers ≈ vCPUs, threads_per_worker=1. - NumPy/Pandas/Numba that release the GIL: - You can use threads; test threads_per_worker > 1. - Memory-heavy tasks: - Set memory_limit per worker (e.g., 70–80% of container memory divided by workers). - Consider Dask’s spilling to disk; it’s safer but slower than OOM. - Don’t oversubscribe: - (#workers × threads_per_worker) >> vCPUs usually hurts for CPU-bound tasks. - Parallelism comes from many small/medium tasks: - Break monoliths; large single tasks won’t benefit from Dask. - If you consistently need more than one node’s worth of resources, use an external cluster and point DaskTaskRunner at it. Example: Local cluster inside one ECS task - ECS task (via work_pool.job_variables) picks the ceiling:

# prefect.yaml (ECS task sizing)
deployments:
- name: etl-heavy
  entrypoint: flows/etl.py:flow
  work_pool:
    name: my-ecs-pool
    job_variables:
      cpu: 4096      # 4 vCPU
      memory: 16384  # 16 GiB
      image: <your-ecr>/my-image:latest

- Flow uses DaskTaskRunner tuned to match:

from prefect import flow, task
from prefect_dask.task_runners import DaskTaskRunner

# 4 vCPU, 16 GiB container => 4 workers, 1 thread each, 3 GiB per worker with headroom
task_runner = DaskTaskRunner(
    cluster_class="distributed.LocalCluster",
    cluster_kwargs={
        "n_workers": 4,
        "threads_per_worker": 1,
        "memory_limit": "3GiB",
        # "processes": True,  # default; explicit if you like
    },
)

@task
def crunch(x):
    # CPU-bound work here
    return x ** 2

@flow(task_runner=task_runner)
def etl():
    return [crunch.submit(i) for i in range(1000)]

Example: Use an external Dask cluster

from prefect import flow, task
from prefect_dask.task_runners import DaskTaskRunner

# Connect to an already-running Dask scheduler (e.g., on K8s or ECS)
task_runner = DaskTaskRunner(address="<tcp://dask-scheduler.default.svc:8786>")

@flow(task_runner=task_runner)
def etl():
    ...

- Here, scale worker CPU/memory on the Dask side; your ECS task for Prefect can be modest. What this means for “auto-sizing” - Prefect won’t auto-pick ECS size or Dask worker size based on runtime intensity. - Good patterns: - Multiple deployments: small/medium/large ECS sizes and Dask config; trigger the right one. - Per-run overrides: pass job_variables (ECS cpu/memory) and environment variables or parameters that affect Dask cluster_kwargs. - External Dask cluster with autoscaling: let Dask (e.g., KubeCluster with adapt) scale workers based on load, while Prefect just orchestrates. Useful docs - Prefect Dask collection: https://prefecthq.github.io/prefect-dask/

- Dask LocalCluster options: https://distributed.dask.org/en/stable/api.html#distributed.LocalCluster - Dask best practices: https://docs.dask.org/en/stable/best-practices.html If you tell me your current ECS task size and the nature of your workload (CPU-bound vs memory-heavy), I’ll give you a concrete n_workers/threads/memory_limit recipe. I live to tune knobs, apparently.

Here are sane, Fargate‑valid presets I see teams use for “small/medium/large.” Pick a profile that matches your workload type. Balanced (general ETL, some Pandas) - Small: 1 vCPU, 4 GiB (cpu=1024, memory=4096) - Medium: 2 vCPU, 8 GiB (cpu=2048, memory=8192) - Large: 4 vCPU, 16 GiB (cpu=4096, memory=16384) CPU‑heavy (lots of pure Python/Numba/compute) - Small: 2 vCPU, 4 GiB (cpu=2048, memory=4096) - Medium: 4 vCPU, 8 GiB (cpu=4096, memory=8192) - Large: 8 vCPU, 16 GiB (cpu=8192, memory=16384) Memory‑heavy (joins, wide DataFrames) - Small: 1 vCPU, 8 GiB (cpu=1024, memory=8192) - Medium: 2 vCPU, 16 GiB (cpu=2048, memory=16384) - Large: 4 vCPU, 32 GiB (cpu=4096, memory=32768) Notes - These respect Fargate’s allowed CPU/memory pairs. If you go off‑menu, AWS will complain. See the matrix: AWS ECS task size combinations. - In Prefect 3.x, set these per deployment via work_pool.job_variables (memory is MiB):

deployments:
- name: myflow-small
  entrypoint: flows/myflow.py:flow
  work_pool:
    name: my-ecs-pool
    job_variables: { cpu: 1024, memory: 4096 }

- name: myflow-medium
  entrypoint: flows/myflow.py:flow
  work_pool:
    name: my-ecs-pool
    job_variables: { cpu: 2048, memory: 8192 }

- name: myflow-large
  entrypoint: flows/myflow.py:flow
  work_pool:
    name: my-ecs-pool
    job_variables: { cpu: 4096, memory: 16384 }

- Docs on overriding per deployment: Customize job variables If you’re using DaskTaskRunner inside the ECS task - A simple rule of thumb: - n_workers ≈ vCPU - threads_per_worker = 1 (for CPU‑bound) - memory_limit per worker ≈ 70–80% of total memory / vCPU - Example for 4 vCPU/16 GiB: n_workers=4, threads_per_worker=1, memory_limit≈"3GiB". - Dask best practices: Dask best practices If you tell me CPU vs memory skew of your flow, I’ll map it to one of the profiles (or give you an XL tier if you like to live dangerously).