We are on the latest prefect amp prefect aws and seeing sche

Question

We are on the latest prefect amp prefect aws and seeing scheduled jobs terminate without being run intermittently The UI for the flow run shows no logs no task runs and no subflow runs In cloudwatch logs for the flow run I found the below Any thoughts ```23 02 20 896 | INFO | prefect engine Engine execution of flow run 682567a2 e507 4aa1 b5c7 25bb2cc2573f aborted by orchestrator This run has already terminated ```

Carlo · Answer

I should say close to the latest ```prefect==2 6 4 prefect aws==0 1 6```

Carlo · Answer

In the agent logs it seems like it timed out submitted the task

Carlo · Answer

Does anyone know of any known issues or settings we need to tune

Christopher Boyd · Answer

It looks like it s failing to submit to the infrastructure as in the agent tries to submit hangs out for 120 seconds and dies because it wasn t successful in submitting

Christopher Boyd · Answer

seems like something going on with the infra side of ECS

Christopher Boyd · Answer

are there enough resources

Carlo · Answer

We are running on Fargate I don t know of limits on task submission there

Christopher Boyd · Answer

What do your agents have for logs The only error I see is that the agent failed to submit to the infrastructure itself because it took too long

Christopher Boyd · Answer

you can probably turn up your log level to debug as well

Carlo · Answer

ok let me try that

Carlo · Answer

appreciate your help

Carlo · Answer

what I posted for logs was all I currently have

Christopher Boyd · Answer

I understand it s just hard to say for sure from those it looks like it tries to submit hangs around before actually posting a job and dies Then from cloudwatch i m not sure if maybe the job actually made it through then tries to communicate to the engine only to find out that it was terminated My suspicion has to do with that initial submission is this intermittent or all jobs some jobs certain times of day I guess i just wonder if it s an issue with fargate and networking based on the error

Carlo · Answer

Your intuition feels right to me It s intermittent but no consistency around timing It could be aws networking but we have been running prefect 1 for almost 2 years w o facing this I will bump up the logs and came back w findings

Christopher Boyd · Answer

It s definitely odd behavior especially if you ve been running for that long

Carlo · Answer

The debug logs provided something concrete to research

Carlo · Answer

Based on amp I think I need to set ```AWS RETRY MODE```

Carlo · Answer

I will try and post back

Christopher Boyd · Answer

good find awesome

Carlo · Answer

So I realized that we had AWS MAX ATTEMPTS set at 10 so I bumped to 100

Carlo · Answer

With that said it looks like this PR was merged to help w this <https github com PrefectHQ prefect pull 5059 files> I believe it includes the flow run id in in the family Which I m not sure is happening now w EcsTask block Perhaps this is why it surfaced in V2

Carlo · Answer

Unfortunately bumping AWS MAX ATTEMPTS didn t help I think prefect aws should include the flow run id in the family

Carlo · Answer

We hot fixed our agent to override this line w the below I ll come back w findings ```task definition setdefault family f prefect self labels task definition family = f task definition family self labels ```

Carlo · Answer

Hey < Christopher Boyd> any thoughts on getting this change pulled into the project

Christopher Boyd · Answer

Hi Carlo I can review the changes and open an issue this week thank you for reporting

Carlo · Answer

Thank you I ll set a reminder to check back middle of next week

Christopher Boyd · Answer

Hey < Carlo> is reported if you would like to add any additional commentary or insight from your experience

Carlo · Answer

Thanks that describes it I will follow issue and comment if I can provide insight