https://prefect.io logo
c

Carlo

11/06/2020, 1:28 PM
šŸ‘‹ I'm running into issues setting up Fargate tasks. The agent successfully launches the ECS task, but the task fails at the validation stage after it loads the flow from S3. It thinks the task and container definitions are different, but neither has changed since I registered the flow and the agent scheduled it. I wonder if it's because the flow is using S3 storage? I didn't use Docker because I couldn't figure out how to use it with ECR and provide credentials. Thoughts?
Copy code
2020-11-06T07:56:19.166-05:00

Copy
	containerDefinition.flow-container.command -> Given: ['/bin/sh', '-c', "python -c 'import prefect; prefect.environments.execution.load_and_run_flow()'"], Expected: ['/bin/sh', '-c', 'prefect execute flow-run']
containerDefinition.flow-container.command -> Given: ['/bin/sh', '-c', "python -c 'import prefect; prefect.environments.execution.load_and_run_flow()'"], Expected: ['/bin/sh', '-c', 'prefect execute flow-run']
a

ale

11/06/2020, 1:35 PM
Hi @Carlo, which version of Prefect are you using? I faced a similar issue with Prefect version < 0.13.13, but it should have been solved in 0.13.13
c

Carlo

11/06/2020, 1:38 PM
I think I'm on an older version than that, let me upgrade. šŸ™ Thank you
a

ale

11/06/2020, 1:39 PM
Let me know how it goes
c

Carlo

11/06/2020, 2:07 PM
I set everything to
0.13.13
but the error still persists. Are you using s3 storage?
a

ale

11/06/2020, 2:14 PM
Nope, I’m using Docker/ECR
However, you error seems to be related to the comparison between the existing task definition and the new task definition. Maybe @Spencer can help
s

Spencer

11/06/2020, 2:16 PM
prefect execute flow-run
would be used by the Agent
The task family that the Fargate Agent uses should be different from the FargateTaskEnvironment task family
c

Carlo

11/06/2020, 2:21 PM
I'm not setting a command explicitly in either the flow fargate task env or the agent
s

Spencer

11/06/2020, 2:21 PM
That's correct
c

Carlo

11/06/2020, 2:21 PM
I do have family set on the flow fargate task env, but not setting one on the agent
s

Spencer

11/06/2020, 2:22 PM
So, the Agent will create an ECS Task Definition which when executed will create another Task Definition to run the flow. My intuition is that these are referencing the same definition for some reason.
c

Carlo

11/06/2020, 2:23 PM
oh, I think it might be because I have taskDefinition and family set to the same value in the fargate task env
let me change this
s

Spencer

11/06/2020, 2:28 PM
Well, in the FargateTaskEnvironment, the
taskDefinition
and
family
are typically the same. I think those values may be the same as the flow-run
taskDefinition
that the Agent is creating. You can look in the AWS ECS console to see the existing
taskDefinition
c

Carlo

11/06/2020, 2:28 PM
This solved it, different errors now, but I think I can fix these
s

Spencer

11/06/2020, 2:28 PM
Oh ok. It seems that there could be a really odd scenario where the
taskDefinition
here is named the same as a flow's flow-run
taskDefinition
would be by the Agent. This probably won't happen if you're not using
enable_task_revisions
since you'll get a fairly unique name, otherwise you could have a flow has a name that would collide. ( https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/fargate/agent.py#L478-L487 )
c

Carlo

11/06/2020, 2:30 PM
I will do some more research and come back
I think it's because I had
flow.name
fargateTaskEnvironment.family
fargateTaskEnvironment.taskDefinition
all set to the same value
šŸ‘ 1
I was able to get my flow working. Thanks for the help, it really unblocked me.