trying to submit a deployment with ECS task overrides as json patches.....here is the command ```pre...

Braun Reyes

04/04/2023, 5:42 PM

trying to submit a deployment with ECS task overrides as json patches.....here is the command

Copy code

prefect deployment build example_flow.py:flow_1 -n base -sb s3/common-utils-dev-test/flow-group-1/flow-1 -ib ecs-task/common-utils-flow-1-dev-test --override 'task_customizations=[{"op": "replace", "path": "/overrides/cpu", "value": "512"}, {"op": "replace", "path": "/overrides/memory", "value": "1024"}]' -q prefect-v2-queue-dev-test -v dev --params '{"param_name": "param_value"}' -t common-utils -t flow-group-1 -a --cron '0 0 * * *'

It results in this infra override in deployments

Copy code

{
  "task_customizations": "[{\"op\": \"replace\", \"path\": \"/overrides/cpu\", \"value\": \"512\"}, {\"op\": \"replace\", \"path\": \"/overrides/memory\", \"value\": \"1024\"}]"
}

however when I submit the flow run I get this error

Copy code

Submission failed. pydantic.error_wrappers.ValidationError: 1 validation error for ECSTask task_customizations instance of JsonPatch expected (type=type_error.arbitrary_type; expected_arbitrary_type=JsonPatch)

anyone know why this would not work.

Braun Reyes

04/04/2023, 7:52 PM

think I have a fix for it This https://github.com/PrefectHQ/prefect-aws/blob/7b64b70cdd2fdb6f8da81b0e2455e41b276ddd59/prefect_aws/ecs.py#L540 needs to be updated to something like this:

Copy code

@validator("task_customizations", pre=True)
    def cast_customizations_to_a_json_patch(
        cls, value: Union[List[Dict], JsonPatch]
    ) -> JsonPatch:
        """
        Casts lists to JsonPatch instances.
        """
        if isinstance(value, str):
            value = json.loads(value)
        if isinstance(value, list):
            return JsonPatch(value)
        return value

✨ 1

Yaron Levi

04/30/2023, 5:13 AM

@Braun Reyes Hi 👋 Have you managed to solve the issue?

Braun Reyes

05/01/2023, 5:49 PM

oh hi yes...my changes were deployed...so things are working now

Yaron Levi

05/02/2023, 10:58 AM

So by using the --override option you were able to set custom CPU and Memory for the Task Definition?

Yaron Levi

05/02/2023, 10:58 AM

(using task_customizations)

Braun Reyes

05/02/2023, 8:39 PM

yes that is correct

Yaron Levi

05/03/2023, 7:12 AM

Thanks!

Yaron Levi

05/03/2023, 7:43 AM

@Braun Reyes Mm… I this exact value that works for you? I am getting this error:

Yaron Levi

05/03/2023, 7:43 AM

Yaron Levi

05/03/2023, 7:43 AM

the override looks like that:

Yaron Levi

05/03/2023, 7:43 AM

Yaron Levi

05/03/2023, 7:44 AM

Any idea what could have gone wrong here?

Yaron Levi

05/03/2023, 7:54 AM

OK (-: Found simple way of doing it:

Yaron Levi

05/03/2023, 7:54 AM

https://discourse.prefect.io/t/dynamically-setting-cpu-memory-and-environment-variables-to-ecstask-block/2866/3

Yaron Levi

05/03/2023, 7:55 AM

Still it’s interesting why the task_customizations worked for you, and I got a crash…

Braun Reyes

05/03/2023, 9:06 PM

Here is mine

Copy code

{
  "task_customizations": "[{\"op\": \"add\", \"path\": \"/overrides/cpu\", \"value\": \"1024\"}, {\"op\": \"add\", \"path\": \"/overrides/memory\", \"value\": \"4096\"}, {\"op\": \"add\", \"path\": \"/networkConfiguration/awsvpcConfiguration/subnets\", \"value\": [\"subnet-xxx\", \"subnet-xxx\", \"subnet-xxx\", \"subnet-xxx\"]}, {\"op\": \"add\", \"path\": \"/networkConfiguration/awsvpcConfiguration/securityGroups\", \"value\": [\"sg-xxx\"]}, {\"op\": \"add\", \"path\": \"/networkConfiguration/awsvpcConfiguration/assignPublicIp\", \"value\": \"DISABLED\"}]"
}

Braun Reyes

05/03/2023, 9:07 PM

i think its just add vs replace

Yaron Levi

05/06/2023, 6:57 PM

Thanks I’ll try it!

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.