Hello, a question about some trouble I'm running into running agent on ECS I followed this recipe: https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-agent. all of the provisions seem to be successful. I can submit a flow run from a deployment, and I see that it is scheduled, but it crashes, and the agent shows these logs:

2023-04-25T17:10:28.843-04:00	21:10:28.842 | INFO | prefect.agent - Completed submission of flow run '01b5b551-18cf-4af5-b08e-072572d7ac6b'

2023-04-25T17:13:35.927-04:00	21:13:35.926 | INFO | prefect.agent - Submitting flow run '938089b4-3452-40a1-9f32-f89a78232497'

2023-04-25T17:13:36.591-04:00	21:13:36.590 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Retrieving task definition 'prefect__timeseries__timeseries2'...

2023-04-25T17:13:36.621-04:00	21:13:36.620 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Registering task definition...

2023-04-25T17:13:37.168-04:00	21:13:37.168 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Creating task run...

2023-04-25T17:13:37.925-04:00	21:13:37.924 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Waiting for task run to start...

2023-04-25T17:13:37.951-04:00	21:13:37.950 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is PROVISIONING.

2023-04-25T17:13:48.030-04:00	21:13:48.029 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is PENDING.

2023-04-25T17:14:03.136-04:00	21:14:03.135 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is RUNNING.

2023-04-25T17:14:08.143-04:00	21:14:08.142 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Running command 'python -m prefect.engine' in container 'prefect' (prefecthq/prefect:latest)...

2023-04-25T17:14:08.155-04:00	21:14:08.155 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Streaming output from container 'prefect'...

2023-04-25T17:14:08.190-04:00	21:14:08.189 | INFO | prefect.agent - Completed submission of flow run '938089b4-3452-40a1-9f32-f89a78232497'

2023-04-25T17:14:08.668-04:00	/usr/local/bin/python: No module named prefect.engine.__main__; 'prefect.engine' is a package and cannot be directly executed

2023-04-25T17:14:13.737-04:00

Copy
21:14:13.736 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is DEPROVISIONING.
21:14:13.736 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is DEPROVISIONING.

2023-04-25T17:14:29.429-04:00	21:14:29.428 | INFO | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Status is STOPPED.

2023-04-25T17:14:29.532-04:00	21:14:29.532 | WARNING | prefect.infrastructure.ecs-task - ECSTask 'mysterious-peccary': Container 'prefect' exited with non-zero exit code 1.

2023-04-25T17:14:29.635-04:00	21:14:29.635 | INFO | prefect.agent - Reported flow run '938089b4-3452-40a1-9f32-f89a78232497' as crashed: Flow run infrastructure exited with non-zero status code 1.

It seems to me like the important line is

No module named prefect.engine.__main__; 'prefect.engine' is a package and cannot be directly executed

But i'm not sure how to debug this and can't find a similar problem online. Does anyone know where I can get help?

I think you need the image to be

2-latest

@Zanie the image defined in terraform vars is: prefecthq/prefect:2-python3.10

You can see it pull a different tag in those logs

@Zanie I'm sorry I'm missing that. Can you point out where?

Right after the status is RUNNING

ohhhh my god thank you

you can set the env var

PREFECT_LOGGING_LEVEL=DEBUG