Hi We re starting to look into a transition to Prefect 2 so

Question

Hi We re starting to look into a transition to Prefect 2 so I have a few likely basic questions I m trying to test out getting a very simple from running with s3 storage and ECS infrastructure I ve gotten to the point where the task successfully launches but I m running into a couple issues I ll detail in the comments

Claire Herdeman · Answer

First I m just running an agent locally on an EC2 instance while I m trying things out The flow successfully launched as an ECS task and I verified that it was in a Running state on the correct cluster However the task seemed to hang and fail to complete so I eventually stopped it The agent logs look like the below ```19 54 19 518 | INFO | prefect agent Submitting flow run a756d8dc 35da 4e64 b4c5 5d04c8c5757f 19 54 19 761 | INFO | prefect infrastructure ecs task ECSTask sensible mackerel Registering task definition 19 54 20 190 | INFO | prefect infrastructure ecs task ECSTask sensible mackerel Creating task run 19 54 20 943 | INFO | prefect infrastructure ecs task ECSTask sensible mackerel Waiting for task run to start 19 54 20 962 | INFO | prefect infrastructure ecs task ECSTask sensible mackerel Status is PROVISIONING 19 54 36 033 | INFO | prefect infrastructure ecs task ECSTask sensible mackerel Status is PENDING 19 55 01 146 | INFO | prefect infrastructure ecs task ECSTask sensible mackerel Status is RUNNING 19 55 06 152 | INFO | prefect infrastructure ecs task ECSTask sensible mackerel Running command python m prefect engine in container prefect xxxxxx 19 55 06 152 | INFO | prefect agent Completed submission of flow run a756d8dc 35da 4e64 b4c5 5d04c8c5757f 19 55 06 158 | INFO | prefect infrastructure ecs task ECSTask sensible mackerel Streaming output from container prefect 20 09 41 075 | INFO | prefect infrastructure ecs task ECSTask sensible mackerel Status is DEPROVISIONING 20 09 56 521 | INFO | prefect infrastructure ecs task ECSTask sensible mackerel Status is STOPPED 20 09 56 691 | WARNING | prefect infrastructure ecs task ECSTask sensible mackerel Container prefect exited with non zero exit code 143 ```

Claire Herdeman · Answer

It seems like the task may not be getting past the ` python m prefect engine ` command Any thoughts there

Claire Herdeman · Answer

Additionally even though the agent says that the flow moved through states PENDING RUNNING DEPROVISIONING and STOPPED the UI still says that the flow is in a Pending state

Anna Geller · Answer

Exit Code 143 means that the container received a SIGTERM signal from the operating system which asks the container to gracefully terminate and the container succeeded in gracefully terminating otherwise you will see Exit Code 137 Q1 are you running this on EC2 or Fargate Q2 Do you want to try <https github com anna geller dataflow ops|this recipe> it has a blog post and video demo in the readme as well

Zanie · Answer

Enabling DEBUG level logs will likely be helpful too `PREFECT DEBUG MODE=1` as an environment variable is a quick way to get there

Claire Herdeman · Answer

Ah yes the Exit Code 143 is because I manually killed the task since it seemed to be hanging See the ~15 mins between the streaming output stmt and the DEPROVISIONING stmt after I cancelled Q1 I m launching the task on Fargate Q2 Yes I am using it I m trying to build up the pieces to make sure I understand how everything fits together

Claire Herdeman · Answer

Re debug level in the ECSTask definition or where the agent is running

Zanie · Answer

In the ECS task

Claire Herdeman · Answer

Ok update with that env var and added some additional networking configuration from the agent ```21 19 43 544 | INFO | prefect agent Submitting flow run a43556de f21f 458d 83f6 559e7b29e41d 21 19 43 790 | INFO | prefect infrastructure ecs task ECSTask ancient macaque Registering task definition 21 19 43 792 | DEBUG | prefect infrastructure ecs task Task definition payload containerDefinitions image xxxx logConfiguration logDriver awslogs options awslogs create group true awslogs group prefect awslogs region us east 2 awslogs stream prefix ancient macaque name prefect cpu 256 executionRoleArn arn aws iam xxxx family prefect memory 512 networkMode awsvpc requiresCompatibilities FARGATE 21 19 44 212 | INFO | prefect infrastructure ecs task ECSTask ancient macaque Creating task run 21 19 44 215 | DEBUG | prefect infrastructure ecs task Task run payload cluster arn aws ecs xxxxxx launchType FARGATE networkConfiguration awsvpcConfiguration assignPublicIp DISABLED securityGroups sg xxxx sg xxxx subnets subnet xxxx subnet xxxx overrides containerOverrides command python m prefect engine cpu 256 environment name PREFECT DEBUG MODE value 1 name PREFECT API URL value name PREFECT API KEY value xxxxx name PREFECT FLOW RUN ID value a43556def21f458d83f6559e7b29e41d memory 512 name prefect cpu 256 executionRoleArn arn aws iam xxxx memory 512 taskRoleArn arn aws iam xxxx tags key value a43556de f21f 458d 83f6 559e7b29e41d key value ancient macaque key value 2 6 4 taskDefinition arn aws ecs xxxxx 21 19 44 849 | INFO | prefect infrastructure ecs task ECSTask ancient macaque Waiting for task run to start 21 19 44 870 | INFO | prefect infrastructure ecs task ECSTask ancient macaque Status is PROVISIONING 21 19 48 550 | DEBUG | prefect agent Checking for flow runs 21 19 53 613 | DEBUG | prefect agent Checking for flow runs 21 19 54 923 | INFO | prefect infrastructure ecs task ECSTask ancient macaque Status is PENDING 21 19 58 755 | DEBUG | prefect agent Checking for flow runs 21 20 03 863 | DEBUG | prefect agent Checking for flow runs 21 20 09 141 | DEBUG | prefect agent Checking for flow runs 21 20 14 279 | DEBUG | prefect agent Checking for flow runs 21 20 19 376 | DEBUG | prefect agent Checking for flow runs 21 20 24 469 | DEBUG | prefect agent Checking for flow runs 21 20 29 953 | DEBUG | prefect agent Checking for flow runs 21 20 30 098 | INFO | prefect infrastructure ecs task ECSTask ancient macaque Status is RUNNING 21 20 35 060 | DEBUG | prefect agent Checking for flow runs 21 20 35 103 | INFO | prefect infrastructure ecs task ECSTask ancient macaque Running command python m prefect engine in container prefect xxxx 21 20 35 103 | INFO | prefect agent Completed submission of flow run a43556de f21f 458d 83f6 559e7b29e41d 21 20 35 109 | INFO | prefect infrastructure ecs task ECSTask ancient macaque Streaming output from container prefect 21 20 40 160 | DEBUG | prefect agent Checking for flow runs 21 20 33 692 | DEBUG | prefect client Connecting to API at 21 20 45 260 | DEBUG | prefect agent Checking for flow runs 21 20 50 354 | DEBUG | prefect agent Checking for flow runs ```

Claire Herdeman · Answer

Then in the cloudwatch logs for the task the only thing so far is ```21 20 33 692 | DEBUG | prefect client Connecting to API at <https api prefect cloud api accounts xxx>```

Zanie · Answer

Sounds like it can t connect I d investigate your network setup You could test it by calling ECSTask run with a ping command or something