👋 I'm running into issues setting up Fargate tasks. The agent successfully launches the ECS task, but the task fails at the validation stage after it loads the flow from S3. It thinks the task and container definitions are different, but neither has changed since I registered the flow and the agent scheduled it. I wonder if it's because the flow is using S3 storage? I didn't use Docker because I couldn't figure out how to use it with ECR and provide credentials. Thoughts?

2020-11-06T07:56:19.166-05:00

Copy
	containerDefinition.flow-container.command -> Given: ['/bin/sh', '-c', "python -c 'import prefect; prefect.environments.execution.load_and_run_flow()'"], Expected: ['/bin/sh', '-c', 'prefect execute flow-run']
containerDefinition.flow-container.command -> Given: ['/bin/sh', '-c', "python -c 'import prefect; prefect.environments.execution.load_and_run_flow()'"], Expected: ['/bin/sh', '-c', 'prefect execute flow-run']

Hi @Carlo, which version of Prefect are you using? I faced a similar issue with Prefect version < 0.13.13, but it should have been solved in 0.13.13

I think I'm on an older version than that, let me upgrade. 🙏 Thank you

Let me know how it goes

I set everything to

0.13.13

but the error still persists. Are you using s3 storage?

Nope, I’m using Docker/ECR

The command in this container (the

flow-container

) is injected by Prefect itself. https://github.com/PrefectHQ/prefect/blob/8563cbb4b0a9e5db99045fb46717cac6ba172af5/src/prefect/environments/execution/fargate/fargate_task.py#L227-L245

I'm not setting a command explicitly in either the flow fargate task env or the agent

That's correct

I do have family set on the flow fargate task env, but not setting one on the agent

So, the Agent will create an ECS Task Definition which when executed will create another Task Definition to run the flow. My intuition is that these are referencing the same definition for some reason.

oh, I think it might be because I have taskDefinition and family set to the same value in the fargate task env

Well, in the FargateTaskEnvironment, the

taskDefinition

and

family

are typically the same. I think those values may be the same as the flow-run

taskDefinition

that the Agent is creating. You can look in the AWS ECS console to see the existing

taskDefinition

This solved it, different errors now, but I think I can fix these

Oh ok. It seems that there could be a really odd scenario where the

taskDefinition

here is named the same as a flow's flow-run

taskDefinition

would be by the Agent. This probably won't happen if you're not using

enable_task_revisions

since you'll get a fairly unique name, otherwise you could have a flow has a name that would collide. ( https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/fargate/agent.py#L478-L487 )

I will do some more research and come back