https://prefect.io logo
Title
n

Nicholas Torba

04/25/2023, 10:01 PM
Hello, a question about some trouble I'm running into running agent on ECS I followed this recipe: https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-agent. all of the provisions seem to be successful. I can submit a flow run from a deployment, and I see that it is scheduled, but it crashes, and the agent shows these logs:
2023-04-25T17:10:28.843-04:00	21:10:28.842 | INFO | prefect.agent - Completed submission of flow run '01b5b551-18cf-4af5-b08e-072572d7ac6b'

2023-04-25T17:13:35.927-04:00	21:13:35.926 | INFO | prefect.agent - Submitting flow run '938089b4-3452-40a1-9f32-f89a78232497'

2023-04-25T17:13:36.591-04:00	21:13:36.590 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Retrieving task definition 'prefect__timeseries__timeseries2'...

2023-04-25T17:13:36.621-04:00	21:13:36.620 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Registering task definition...

2023-04-25T17:13:37.168-04:00	21:13:37.168 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Creating task run...

2023-04-25T17:13:37.925-04:00	21:13:37.924 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Waiting for task run to start...

2023-04-25T17:13:37.951-04:00	21:13:37.950 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is PROVISIONING.

2023-04-25T17:13:48.030-04:00	21:13:48.029 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is PENDING.

2023-04-25T17:14:03.136-04:00	21:14:03.135 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is RUNNING.

2023-04-25T17:14:08.143-04:00	21:14:08.142 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Running command 'python -m prefect.engine' in container 'prefect' (prefecthq/prefect:latest)...

2023-04-25T17:14:08.155-04:00	21:14:08.155 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Streaming output from container 'prefect'...

2023-04-25T17:14:08.190-04:00	21:14:08.189 | INFO | prefect.agent - Completed submission of flow run '938089b4-3452-40a1-9f32-f89a78232497'

2023-04-25T17:14:08.668-04:00	/usr/local/bin/python: No module named prefect.engine.__main__; 'prefect.engine' is a package and cannot be directly executed

2023-04-25T17:14:13.737-04:00

Copy
21:14:13.736 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is DEPROVISIONING.
21:14:13.736 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is DEPROVISIONING.

2023-04-25T17:14:29.429-04:00	21:14:29.428 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is STOPPED.

2023-04-25T17:14:29.532-04:00	21:14:29.532 | WARNING | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Container 'prefect' exited with non-zero exit code 1.

2023-04-25T17:14:29.635-04:00	21:14:29.635 | INFO | prefect.agent - Reported flow run '938089b4-3452-40a1-9f32-f89a78232497' as crashed: Flow run infrastructure exited with non-zero status code 1.
It seems to me like the important line is
No module named prefect.engine.__main__; 'prefect.engine' is a package and cannot be directly executed
But i'm not sure how to debug this and can't find a similar problem online. Does anyone know where I can get help?
z

Zanie

04/25/2023, 11:25 PM
I think you need the image to be
2-latest
Looks like it tried to run a prefect 1 image which doesn't have the same entrypoint for the engine
n

Nicholas Torba

04/25/2023, 11:41 PM
@Zanie the image defined in terraform vars is: prefecthq/prefect:2-python3.10
z

Zanie

04/26/2023, 12:14 AM
You can see it pull a different tag in those logs
n

Nicholas Torba

04/26/2023, 12:20 AM
@Zanie I'm sorry I'm missing that. Can you point out where?
z

Zanie

04/26/2023, 12:22 AM
Right after the status is RUNNING
You should also be able to see the image used in the AWS dashboard
We'll also display the task definition used if you turn in debug level logs
n

Nicholas Torba

04/26/2023, 12:27 AM
ohhhh my god thank you
how do I turn on debug level logs?
z

Zanie

04/26/2023, 12:29 AM
you can set the env var
PREFECT_LOGGING_LEVEL=DEBUG