I've recently transitioned my stack from using an Agent to trigger flows over Amazon ECS to an ECS push work pool and am encountering some unexplained issues. During development work I was (eventually) able to trigger flow runs on demand over Fargate infrastructure. without issue. I'm now finding that runs set up on a schedule mostly (~75% of the time) fail to provision the requisite infrastructure, failing with a message saying

Flow run infrastructure exited with non-zero status code:
 Exited with non 0 code. (Error Code: 1)
This may be caused by attempting to run an image with a misspecified platform or architecture.

Oddly, If I manually trigger the same flow runs they work without issue. Because push work pools don't provide logs and logs only go to Cloudwatch after infrastructure is provisioned it's proving hard to troubleshoot this issue. I remember @Will Raphaelson mentioning that the

exited with non 0 code

error was a particular bugbear and you all at @Prefect have experience troubleshooting it. Any ideas what could be causing my issues? I'm using Prefect 2.18.0 over AWS ECS (Fargate) infrastructure.

are you deploying with

prefect deploy

in the cli with a

prefect.yaml

file or

flow.deploy()

in python? and in either case, are you using those strategies to build docker images?

We're using

flow.deploy()

in python. We pull existing AWS Task Definitions which reference docker images we've already prepared and populated to AWS ECR. We are building over Linux boxes as part of a Github CI/CD chain -- not an M1-M3 Mac.

I wonder then if the wrong image is being used? I think you can check the image name on the ecs task itself, then maybe you can track it down and verify it's built for the right platform?

This doesn't really make any sense -- why would the wrong image be used for scheduled runs but not manually triggered runs? More pertinently we only really use one image for our stack, and it's not built on an ARM machine. If I look at the Task Definition in question for safety's sake it's referencing the image I want.

Yeah, you're right. I have another suspicion.

hmmm

you should be able to create a new work queue in your push pool with a different name, and move your deployments over to it

Interesting. I"ll set this up for a few test runs and see if they pass. I'll report back here

going to follow up in dms

thanks!

@Kevin Grismore Any resolution? I'm attempting to deploy an ECS work pool and keep running into a similar error-- each time I deploy from my server my infrastructure provisions and runs but I'm unable to get past "Flow run infrastructure exited with non-zero status code 1." I'm pushing to ECR from an M3 mac, but I'm using act and am running it with the flag

--container-architecture linux/amd64

and have

ENV DOCKER_DEFAULT_PLATFORM=linux/amd64

in my dockerfile