< Marvin> this is how i run flow via rest api it flow runs i Prefect Community #ask-marvin

<@ULVA73B9P> this is how i run flow via rest api ,...

Madushan Pathirana

11/13/2025, 5:39 AM

@Marvin this is how i run flow via rest api , it flow runs in ecs worker . how to overide the memory and cpu and api in the deployemnt yaml work_pool: name: my-ecs-pool work_queue_name: default job_variables: image: 010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{ git_sha.stdout }} # launch_type: EC2 cpu: 4096 # vCPU memory: 8192 # MiB push: false env: APP_ENV: "{{ prefect.variables.app_env }}" SOURCE : "ebay" def get_deployment_id(flow_name: str, deployment_name: str) -> str: prefect_credentials =get_credentials('prefect/prod') USERNAME = prefect_credentials['user'] PASSWORD = prefect_credentials['password'] url = f"{PREFECT_API_URL}/deployments/name/{quote(flow_name)}/{quote(deployment_name)}" r = requests.get(url, auth=(USERNAME, PASSWORD)) r.raise_for_status() data = r.json() # API returns the full deployment object; id is under 'id' or 'deployment_id' depending on version return data.get("id") or data["deployment_id"] def create_flow_run_from_deployment(deployment_id: str, parameters: dict) -> dict: prefect_credentials =get_credentials('prefect/prod') USERNAME = prefect_credentials['user'] PASSWORD = prefect_credentials['password'] url = f"{PREFECT_API_URL}/deployments/{deployment_id}/create_flow_run" payload = {"parameters": parameters} r = requests.post(url, json=payload, auth=(USERNAME, PASSWORD)) r.raise_for_status() return r.json() def run_deployment_by_full_name(flow_name, deployment_name, parameters): dep_id = get_deployment_id(flow_name, deployment_name) print(dep_id) return create_flow_run_from_deployment(dep_id, parameters)

Marvin

11/13/2025, 5:40 AM

✅ thought for 367.4 seconds

Marvin

11/13/2025, 5:46 AM

Great question. In Prefect 3.x (and with the ECS worker), you set defaults for CPU/memory in your deployment’s work pool job_variables, and you can override them per run via the REST API. Key points: - Precedence: flow-run overrides > deployment job_variables > work pool defaults - ECS CPU is in CPU units (1024 = 1 vCPU). Memory is in MiB. - For EC2 launch type, include

launch_type: EC2

. Typical fields also include

cluster

subnets

, and

security_groups

if you’re using awsvpc networking. 1) Set defaults in your deployment YAML Here’s a minimal example matching your setup (ECS EC2 worker):

Copy code

work_pool:
  name: my-ecs-pool
  work_queue_name: default
  job_variables:
    launch_type: EC2
    image: <http://010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{|010614160260.dkr.ecr.ap-southeast-1.amazonaws.com/credilinq-prefect:{{> git_sha.stdout }}
    cpu: 4096     # 4 vCPU
    memory: 8192  # 8 GiB
    push: false
    env:
      APP_ENV: "{{ prefect.variables.app_env }}"
      SOURCE: "ebay"
    # Optional but commonly needed for EC2 + awsvpc:
    # cluster: my-ecs-cluster
    # subnets:
    #   - subnet-xxxxxxxx
    # security_groups:
    #   - sg-xxxxxxxx

2) Override at run time via REST API Add a

job_variables

object to your POST body for

/deployments/{id}/create_flow_run

. In your code, that’s just adding the key to `payload`:

Copy code

def create_flow_run_from_deployment(deployment_id: str, parameters: dict) -> dict:
    prefect_credentials = get_credentials('prefect/prod')
    USERNAME = prefect_credentials['user']
    PASSWORD = prefect_credentials['password']

    url = f"{PREFECT_API_URL}/deployments/{deployment_id}/create_flow_run"
    payload = {
      "parameters": parameters,
      "job_variables": {
        "cpu": 2048,     # 2 vCPU for this run
        "memory": 4096   # 4 GiB for this run
        # you can also override launch_type, image, env, etc. here if needed
        # "launch_type": "EC2"
      }
    }
    r = <http://requests.post|requests.post>(url, json=payload, auth=(USERNAME, PASSWORD))
    r.raise_for_status()
    return r.json()

Common ECS job_variables you can set - cpu, memory, launch_type (EC2 or FARGATE) - image, cluster, container_name - task_role_arn, execution_role_arn - subnets, security_groups, assign_public_ip - env, command - capacity_provider_strategy, platform_version, propagate_tags, enable_execute_command, ephemeral_storage Docs for reference: - Customize job variables (precedence and overrides) - ECS worker guide - prefect.yaml reference - Work pools concept Quick clarification: when you said “and api in the deployment yaml,” did you mean “and via API”? If you meant something else (e.g., a specific API-related setting in YAML), let me know and I’ll tailor the example.

3 Views

Open in Slack

Previous Next