I’m getting an error in my agent logs, using an ECSTask Infrastructure Block ```An error occured whi...

Alex Turek

12/30/2022, 9:51 PM

I’m getting an error in my agent logs, using an ECSTask Infrastructure Block

Copy code

An error occured while monitoring flow run '793a0ee8-fe06-48d3-8858-31b2e99e4c50'. The flow run will not be marked as failed, but an issue may have occurred.

I think the Agent is trying to watch the created ECS Task’s logs, and using the wrong parameters to watch them

Alex Turek

12/30/2022, 9:51 PM

here’s the line of code in

prefect-aws

that shows up in the stack trace https://github.com/PrefectHQ/prefect-aws/blob/267d973fcf9f9ccf1a4508140daf5b033a53c609/prefect_aws/ecs.py#L1013

Alex Turek

12/30/2022, 9:53 PM

the ECSTask is configured like

Copy code

ECSTask(
  ...
  stream_output = True,
  configure_cloudwatch_logs = True,
)

Alex Turek

12/30/2022, 9:53 PM

and the error that’s at the top (?) of the stack trace:

Copy code

An error occurred (ResourceNotFoundException) when calling the GetLogEvents operation: The specified log stream does not exist.

Abosede Oladipo

12/31/2022, 11:11 PM

I'm getting the same error. It looks like the

prefect-aws

code is being built using

log-stream-prefix/family/task-definition:revision

Copy code

stream_name = "/".join(
                    [
                        log_config["awslogs-stream-prefix"],
                        task_definition["family"],
                        task_arn.rsplit("/")[-1],
                    ]
                )

Meanwhile, aws creates the logs stream using

log-stream-prefix/container-name/container-id

Abosede Oladipo

12/31/2022, 11:19 PM

I'm getting the same error. It looks like the

prefect-aws

code is being built using

log-stream-prefix/family/task-definition:revision

Copy code

stream_name = "/".join(
                    [
                        log_config["awslogs-stream-prefix"],
                        task_definition["family"],
                        task_arn.rsplit("/")[-1],
                    ]
                )

Meanwhile, aws docs creates the log stream using

log-stream-prefix/container-name/ecs-task-id

👀 1

Zanie

01/03/2023, 5:05 PM

Can you please open an issue?

Zanie

01/03/2023, 5:05 PM

We have not been able to reproduce this and the tests are working — it’d be great to have a simple example so we can fix this.

Zanie

01/03/2023, 5:52 PM

More details at https://github.com/PrefectHQ/prefect-aws/issues/177 would be appreciated.

👍 1

Zanie

01/04/2023, 6:19 PM

Thanks! https://github.com/PrefectHQ/prefect-aws/pull/186

💃 2

48 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.