https://prefect.io logo
Title
b

Ben Muller

11/02/2022, 9:22 PM
Starting a new thread for a new topic - Are we able to tag infrastructure blocks? I would like to tag my EcsTask but can not see the arg available for it in the code?
1
k

Kalise Richmond

11/02/2022, 9:24 PM
Just curious, what are you trying to do with a tag on a block? Is this for UI searchability or filtering the blocks page?
b

Ben Muller

11/02/2022, 9:28 PM
yep - I want it for searchability - to give some more info - and correct me if I am mislead in anyway. The way I see it, flows have a
one:many
relationship with deployments, and deployments ( maybe only in the ECSTask space ) have a
one:one
relationship with infrastructure blocks. I know this is probably not by design, but the reality is my cpu and memory requirements will change according to the deployment I am running, and it is probably easier to have one infrastructure block per deployment than to locate and identify which block suits which deployment. So the reason I want tags on an infra block is when a deployment is no longer needed ( in my head a schedule may become redundant ) - I want the ability to look at my deployments tags and then locate the infrastructure block too so I can delete it. Am I thinking in the right way ?
or is there in fact a way to have one infrastructure block for ECS and change the memory and cpu requirements on the fly when the flow runs? That would be the dream for me, but I see you have to peg infrastructure to a deployment at build time, so this is not possible
z

Zanie

11/02/2022, 9:45 PM
I think that is the intention of the
infra_overrides
option on deployments.
b

Ben Muller

11/02/2022, 9:46 PM
AHhhhhh how good
z

Zanie

11/02/2022, 9:46 PM
The implementation of that needs a little love, but it should work as you desire. The agent updates a local copy of the block with the contents there before using it for submission.
b

Ben Muller

11/02/2022, 9:46 PM
so I just need to provide a dict with cpu and memory as the keys?
that makes life a lot more pleasant ! thanks for pointing that out
z

Zanie

11/02/2022, 9:52 PM
Yeah if it doesn’t behave let us know! It might bypass the validators or something as the implementation just recursively sets items.
b

Ben Muller

11/02/2022, 9:53 PM
yeah - I will give it a shot now and let you know
Worked perfectly - thanks for the steer @Zanie - that is a huge saving of duplication 😛arty-parrot:
last question - I can see that the name of the
taskDefinition
being ran in ECS is defaulting to
prefect
. This will cause some issues when multiple ECS blocks are running with different images. How can I change this name? Do I actually need to provide an entire container definition for this to be changed?
@Zanie - I can see it is being set here But can not work out how I can change that. Do I need to provide an empty task def dict with containerDefinitions ?
I think this would cause issues, as I remember it in 1.0 prefect would create a new task definition for every flow ?
z

Zanie

11/03/2022, 3:23 PM
The container name itself must be
prefect
so that we can find the container to run the flow in
The task definition family name can be something else, but we haven’t hooked up customization of that yet.
We do need to create a new task definition for every flow because some settings can only be changed at the task definition level. If you run into rate limiting issues, you can register your own task definition and point us to it. Then, we will only update it if you ask for settings that are different than it has https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/ecs.py#L523-L530
b

Ben Muller

11/03/2022, 5:43 PM
I just remember with Prefect 1 that the task def name in aws had the flow name in it. If I have multiple repos that have flows (which I do) and the use different custom images. What's going to happen when two flows from different repos are scheduled at the same time with the same name etc. Might be challenging to also see which task is which in ecs. Prefect 1 had an issue where sometimes they weren't terminated properly.
z

Zanie

11/03/2022, 5:44 PM
Yeah we don’t have access to the flow name in
Infrastructure.run
but it’s looking like it’d be nice so we’ll need to find a way to pass it through
:marvin: 1
👍 1
b

Ben Muller

11/08/2022, 7:41 AM
Hey @Zanie - as I thought, I have started our migration from 1.0 to 2.0 and we have a repo with 10 flows, all flows in the repo share the same base image, and same
ecs-task
block. Now that they are in prod and when any two flows share the same schedule ( eg
0 * * * *
). I am getting hit with an error from prefect
State message: Submission failed. botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
This is a pretty big deal for our migration, how can I get around this ? really hoping there is something we can do
You can also add a customization to override the family name per deployment
You can also configure AWS client retries in your agent’s environment
b

Ben Muller

11/09/2022, 6:28 PM
Wow wow wow. @Zanie this is perfect. Thank you!
bow
Judy confirming my understanding. This would mean I would have one ecstask block per deployment or flow. Those labels can't be set at run time?
z

Zanie

11/09/2022, 6:54 PM
The agent sets the labels at runtime by calling
infrastructure.prepare_for_flow_run
b

Ben Muller

11/09/2022, 7:32 PM
Amazing!