Starting a new thread for a new topic Are we able to tag inf

Question

Starting a new thread for a new topic Are we able to tag infrastructure blocks I would like to tag my EcsTask but can not see the arg available for it in the code

Kalise Richmond · Answer

Just curious what are you trying to do with a tag on a block Is this for UI searchability or filtering the blocks page

Ben Muller · Answer

yep I want it for searchability to give some more info and correct me if I am mislead in anyway The way I see it flows have a `one many` relationship with deployments and deployments maybe only in the ECSTask space have a `one one` relationship with infrastructure blocks I know this is probably not by design but the reality is my cpu and memory requirements will change according to the deployment I am running and it is probably easier to have one infrastructure block per deployment than to locate and identify which block suits which deployment So the reason I want tags on an infra block is when a deployment is no longer needed in my head a schedule may become redundant I want the ability to look at my deployments tags and then locate the infrastructure block too so I can delete it Am I thinking in the right way

Ben Muller · Answer

or is there in fact a way to have one infrastructure block for ECS and change the memory and cpu requirements on the fly when the flow runs That would be the dream for me but I see you have to peg infrastructure to a deployment at build time so this is not possible

Zanie · Answer

I think that is the intention of the `infra overrides` option on deployments

Ben Muller · Answer

AHhhhhh how good

Zanie · Answer

The implementation of that needs a little love but it should work as you desire The agent updates a local copy of the block with the contents there before using it for submission

Ben Muller · Answer

so I just need to provide a dict with cpu and memory as the keys

Ben Muller · Answer

that makes life a lot more pleasant thanks for pointing that out

Zanie · Answer

Yeah if it doesn t behave let us know It might bypass the validators or something as the implementation just recursively sets items

Ben Muller · Answer

yeah I will give it a shot now and let you know

Ben Muller · Answer

Worked perfectly thanks for the steer < Zanie> that is a huge saving of duplication party parrot

Ben Muller · Answer

last question I can see that the name of the `taskDefinition` being ran in ECS is defaulting to `prefect` This will cause some issues when multiple ECS blocks are running with different images How can I change this name Do I actually need to provide an entire container definition for this to be changed

Ben Muller · Answer

< Zanie> I can see it is being set But can not work out how I can change that Do I need to provide an empty task def dict with containerDefinitions

Ben Muller · Answer

I think this would cause issues as I remember it in 1 0 prefect would create a new task definition for every flow

Zanie · Answer

I think you re actually looking for <https github com PrefectHQ prefect aws blob main prefect aws ecs py L949>

Zanie · Answer

The container name itself must be `prefect` so that we can find the container to run the flow in

Zanie · Answer

The task definition family name can be something else but we haven t hooked up customization of that yet

Zanie · Answer

We do need to create a new task definition for every flow because some settings can only be changed at the task definition level If you run into rate limiting issues you can register your own task definition and point us to it Then we will only update it if you ask for settings that are different than it has <https github com PrefectHQ prefect aws blob main prefect aws ecs py L523 L530>

Ben Muller · Answer

I just remember with Prefect 1 that the task def name in aws had the flow name in it If I have multiple repos that have flows which I do and the use different custom images What s going to happen when two flows from different repos are scheduled at the same time with the same name etc Might be challenging to also see which task is which in ecs Prefect 1 had an issue where sometimes they weren t terminated properly

Zanie · Answer

Yeah we don t have access to the flow name in `Infrastructure run` but it s looking like it d be nice so we ll need to find a way to pass it through

Ben Muller · Answer

Hey < Zanie> as I thought I have started our migration from 1 0 to 2 0 and we have a repo with 10 flows all flows in the repo share the same base image and same `ecs task` block Now that they are in prod and when any two flows share the same schedule eg `0 ` I am getting hit with an error from prefect ```State message Submission failed botocore errorfactory ClientException An error occurred ClientException when calling the RegisterTaskDefinition operation Too many concurrent attempts to create a new revision of the specified family ``` This is a pretty big deal for our migration how can I get around this really hoping there is something we can do

Zanie · Answer

Zanie · Answer

You can also add a customization to override the family name per deployment

Zanie · Answer

You can also configure AWS client retries in your agent s environment

Ben Muller · Answer

Wow wow wow < Zanie> this is perfect Thank you

Ben Muller · Answer

bow

Ben Muller · Answer

Judy confirming my understanding This would mean I would have one ecstask block per deployment or flow Those labels can t be set at run time

Zanie · Answer

The agent sets the labels at runtime by calling `infrastructure prepare for flow run`

Zanie · Answer

See <https github com PrefectHQ prefect pull 7479>

Ben Muller · Answer

Amazing