https://prefect.io logo
Title
d

Daniel Bernstein

12/14/2022, 1:56 PM
Hey all, I'm having some issues with the ECS task block. Tasks are launching properly but are then hanging for a half hour before executing the flow run. I am not sure if the hanging issue is in the task or the agent. I am running Prefect 2.7.0 and using prefect-aws 0.2.1. I am confused because, in my dev environment (running Prefect 2.6.7 and prefect-aws 0.1.8) the workflows deploy and run properly. Not sure if this is related, but I am also seeing in prod that the prefect agent cannot identify the log stream for new tasks, returning the below error:
botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the GetLogEvents operation: The specified log stream does not exist.
however, the task is generating logs that appear in AWS CloudWatch. Maybe the agent is looking for logs under a different name than the task is outputting them to? I prefer not to revert back to the early versions that appear to be working in the dev environment but I cannot seem to get the newer versions of prefect and prefect-aws to function as expected. @Taylor Curran
z

Zanie

12/14/2022, 3:36 PM
Can you please open an issue? I think this is just an intermittent error where AWS claims the resource does not exist.
👍 1
c

Chris Gunderson

12/14/2022, 5:18 PM
@Zanie I am seeing this issue too. Skipped other parts of yaml file besides logging. The flow is failing too. It looks like Prefect2 is looking in /usr/local/lib/python3.9/site-packages/ instead of /opt/pysetup/.venv where I specified the virtual environment Parameters: project: Type: String Description: Project name Default: dev-ecsworker Resources: PrefectLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Ref project RetentionInDays: 7 PrefectTaskDefinition: Type: AWS::ECS::TaskDefinition Properties: ContainerDefinitions: LogConfiguration: LogDriver: awslogs Options: awslogs-region: !Ref AWS::Region awslogs-group: !Ref PrefectLogGroup awslogs-stream-prefix: !Ref project
a

Anna Geller

12/14/2022, 7:01 PM
+1 to create an issue with a reproducible example, and if you create one, could you share it here @Daniel Bernstein? thanks in advance
c

Chris Gunderson

12/14/2022, 7:04 PM
Could this be due to an additional task definition being created called Prefect?
a

Anna Geller

12/14/2022, 7:05 PM
Chris, there's so much you can configure here, hard to say without a full example
👍 1
d

Daniel Bernstein

12/14/2022, 9:03 PM
@Chris Gunderson if you would like to put in the issue for the log issue, please do so. It will take me some time to put together a reproducible example and I'm not even sure that the log issue is the core issue I'm trying to fix.
🙌 2
c

Chris Gunderson

12/14/2022, 9:06 PM
@Daniel Bernstein The issue for me was that Prefect was creating a second task definition in the ECS Service. I'm still working to see if I can resolve it on my end by configuration.
:gratitude-thank-you: 2
💪 1
m

Mateo Merlo

01/02/2023, 12:27 PM
Hi @Chris Gunderson, were you able to resolve this? I'm having the same issue with Prefect creating a second task definition.
c

Chris Gunderson

01/03/2023, 1:53 PM
@Mateo Merlo I'm actually just creating the agent now and prefect is creating the container to run the flows. This works and will not cause any performance issues. Our service quota is set to like 5000, so it shouldn't be a problem. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-quotas.html
m

Mateo Merlo

01/03/2023, 7:27 PM
But is still prefect creating a new task definition every time you run a flow? I have a agent running as well, but the flows are running on a different instance with a custom docker image
c

Chris Gunderson

01/03/2023, 7:33 PM
Yes, there is a new task definition created for each new flow that is run.
I pushed an image to ECR and put that image in the Prefect ECS Task. I believe it is using the image specified from ECR.
m

Mateo Merlo

01/03/2023, 8:38 PM
Great, I just wanted to confirm the way is working is correct.
Thanks Chris!
c

Chris Gunderson

01/03/2023, 8:38 PM
You're welcome
t

Taylor Curran

01/03/2023, 9:19 PM
Hi Friends, this PR should fix the
new task definition created for each new flow
issue: https://github.com/PrefectHQ/prefect-aws/pull/181
🙌 1
z

Zanie

01/03/2023, 9:22 PM
Please note that adds caching for the latest task definition but you can and should configure your ECS task to avoid reregistration if you are worried about rate limits.
:thank-you: 1
m

Mateo Merlo

01/03/2023, 9:31 PM
Thanks for the update @Taylor Curran @Zanie!