Hey all, I'm having some issues with the ECS task block. Tasks are launching properly but are then hanging for a half hour before executing the flow run. I am not sure if the hanging issue is in the task or the agent. I am running Prefect 2.7.0 and using prefect-aws 0.2.1. I am confused because, in my dev environment (running Prefect 2.6.7 and prefect-aws 0.1.8) the workflows deploy and run properly. Not sure if this is related, but I am also seeing in prod that the prefect agent cannot identify the log stream for new tasks, returning the below error:

botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the GetLogEvents operation: The specified log stream does not exist.

however, the task is generating logs that appear in AWS CloudWatch. Maybe the agent is looking for logs under a different name than the task is outputting them to? I prefer not to revert back to the early versions that appear to be working in the dev environment but I cannot seem to get the newer versions of prefect and prefect-aws to function as expected. @Taylor Curran

Can you please open an issue? I think this is just an intermittent error where AWS claims the resource does not exist.

@Zanie I am seeing this issue too. Skipped other parts of yaml file besides logging. The flow is failing too. It looks like Prefect2 is looking in /usr/local/lib/python3.9/site-packages/ instead of /opt/pysetup/.venv where I specified the virtual environment Parameters: project: Type: String Description: Project name Default: dev-ecsworker Resources: PrefectLogGroup: Type: AWS:Logs:LogGroup Properties: LogGroupName: !Ref project RetentionInDays: 7 PrefectTaskDefinition: Type: AWS:ECS:TaskDefinition Properties: ContainerDefinitions: LogConfiguration: LogDriver: awslogs Options: awslogs-region: !Ref AWS::Region awslogs-group: !Ref PrefectLogGroup awslogs-stream-prefix: !Ref project

+1 to create an issue with a reproducible example, and if you create one, could you share it here @Daniel Bernstein? thanks in advance

Could this be due to an additional task definition being created called Prefect?

Chris, there's so much you can configure here, hard to say without a full example

@Chris Gunderson if you would like to put in the issue for the log issue, please do so. It will take me some time to put together a reproducible example and I'm not even sure that the log issue is the core issue I'm trying to fix.

@Daniel Bernstein The issue for me was that Prefect was creating a second task definition in the ECS Service. I'm still working to see if I can resolve it on my end by configuration.

Hi @Chris Gunderson, were you able to resolve this? I'm having the same issue with Prefect creating a second task definition.

@Mateo Merlo I'm actually just creating the agent now and prefect is creating the container to run the flows. This works and will not cause any performance issues. Our service quota is set to like 5000, so it shouldn't be a problem. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-quotas.html

But is still prefect creating a new task definition every time you run a flow? I have a agent running as well, but the flows are running on a different instance with a custom docker image

Yes, there is a new task definition created for each new flow that is run.

Great, I just wanted to confirm the way is working is correct.

You're welcome

Hi Friends, this PR should fix the

new task definition created for each new flow

issue: https://github.com/PrefectHQ/prefect-aws/pull/181

Please note that adds caching for the latest task definition but you can and should configure your ECS task to avoid reregistration if you are worried about rate limits.

Thanks for the update @Taylor Curran @Zanie!