< Marvin> I m using ECS as the work pool and using EC2 as th Prefect Community #ask-marvin

<@ULVA73B9P> - I’m using ECS as the work-pool and ...

Frances Liu

10/20/2023, 4:06 AM

@Marvin - I’m using ECS as the work-pool and using EC2 as the agent, and ECR for a custom docker image with all the dependencies installed. Why are new tasks crashing without logs?

Nate

10/20/2023, 4:12 AM

hi @Frances Liu - can you share your agent logs from that time? sorry, openai is having an outage right now!

Copy code

openai.error.APIError: Internal server error {
2023-10-20 05:06:46.511
    "error": {
        "message": "Internal server error",
        "type": "auth_subrequest_error",
        "param": null,
        "code": "internal_error"
    }
}

@Marvin might be a little flaky in the meantime

Frances Liu

10/20/2023, 4:13 AM

Copy code

Oct 20 03:46:25 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 03:46:25.142 | INFO  | prefect.agent - Reported flow run '7bdef44e-37d8-44e1-a679-d9c49e682e27' as crashed: Flow run infrastructure exited with non-zero status code 1.
Oct 20 04:03:31 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:03:31.582 | INFO  | prefect.agent - Submitting flow run '8733eb86-f601-4281-ab61-5a45af63e638'
Oct 20 04:03:32 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:03:32.219 | INFO  | prefect.infrastructure.ecs-task - ECSTask 'belligerent-whale': Retrieving task definition 'prefect__update_ctgov_studies__ecs-task-deployment'...
Oct 20 04:03:32 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:03:32.297 | INFO  | prefect.infrastructure.ecs-task - ECSTask 'belligerent-whale': Registering task definition...
Oct 20 04:03:32 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:03:32.666 | INFO  | prefect.infrastructure.ecs-task - ECSTask 'belligerent-whale': Creating task run...
Oct 20 04:03:33 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:03:33.330 | INFO  | prefect.infrastructure.ecs-task - ECSTask 'belligerent-whale': Waiting for task run to start...
Oct 20 04:03:33 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:03:33.363 | INFO  | prefect.infrastructure.ecs-task - ECSTask 'belligerent-whale': Status is PROVISIONING.
Oct 20 04:03:43 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:03:43.424 | INFO  | prefect.infrastructure.ecs-task - ECSTask 'belligerent-whale': Status is PENDING.
Oct 20 04:04:28 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:04:28.700 | INFO  | prefect.infrastructure.ecs-task - ECSTask 'belligerent-whale': Status is RUNNING.
Oct 20 04:04:33 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:04:33.707 | INFO  | prefect.infrastructure.ecs-task - ECSTask 'belligerent-whale': Running command 'python -m prefect.engine' in container 'prefect' (<http://005812453855.dkr.ecr.us-east-2.amazonaws.com/ecr-prefect:latest|005812453855.dkr.ecr.us-east-2.amazonaws.com/ecr-prefect:latest>)...
Oct 20 04:04:33 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:04:33.797 | INFO  | prefect.agent - Completed submission of flow run '8733eb86-f601-4281-ab61-5a45af63e638'
Oct 20 04:04:38 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:04:38.763 | INFO  | prefect.infrastructure.ecs-task - ECSTask 'belligerent-whale': Status is DEPROVISIONING.
Oct 20 04:04:53 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:04:53.857 | INFO  | prefect.infrastructure.ecs-task - ECSTask 'belligerent-whale': Status is STOPPED.
Oct 20 04:04:53 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:04:53.887 | WARNING | prefect.infrastructure.ecs-task - ECSTask 'belligerent-whale': Container 'prefect' exited with non-zero exit code 1.
Oct 20 04:04:54 ip-172-31-18-51.us-east-2.compute.internal prefect[3795]: 04:04:54.038 | INFO  | prefect.agent - Reported flow run '8733eb86-f601-4281-ab61-5a45af63e638' as crashed: Flow run infrastructure exited with non-zero status code 1.

Frances Liu

10/20/2023, 4:15 AM

@Nate - no worries. happy to get a human 😄

Nate

10/20/2023, 4:34 AM

hmm, for an infrastructure crash I'd probably go looking in the ECS Console for the logs on this

belligerent-whale

container there could be something like a dependency problem thats crashing things

Open in Slack

Previous Next