Thread
#prefect-community
    Carlo

    Carlo

    1 year ago
    šŸ‘‹ I'm running into issues setting up Fargate tasks. The agent successfully launches the ECS task, but the task fails at the validation stage after it loads the flow from S3. It thinks the task and container definitions are different, but neither has changed since I registered the flow and the agent scheduled it. I wonder if it's because the flow is using S3 storage? I didn't use Docker because I couldn't figure out how to use it with ECR and provide credentials. Thoughts?
    2020-11-06T07:56:19.166-05:00
    
    Copy
    	containerDefinition.flow-container.command -> Given: ['/bin/sh', '-c', "python -c 'import prefect; prefect.environments.execution.load_and_run_flow()'"], Expected: ['/bin/sh', '-c', 'prefect execute flow-run']
    containerDefinition.flow-container.command -> Given: ['/bin/sh', '-c', "python -c 'import prefect; prefect.environments.execution.load_and_run_flow()'"], Expected: ['/bin/sh', '-c', 'prefect execute flow-run']
    ale

    ale

    1 year ago
    Hi @Carlo, which version of Prefect are you using? I faced a similar issue with Prefect version < 0.13.13, but it should have been solved in 0.13.13
    Carlo

    Carlo

    1 year ago
    I think I'm on an older version than that, let me upgrade. šŸ™ Thank you
    ale

    ale

    1 year ago
    Let me know how it goes
    Carlo

    Carlo

    1 year ago
    I set everything to
    0.13.13
    but the error still persists. Are you using s3 storage?
    ale

    ale

    1 year ago
    Nope, Iā€™m using Docker/ECR
    However, you error seems to be related to the comparison between the existing task definition and the new task definition. Maybe @Spencer can help
    s

    Spencer

    1 year ago
    prefect execute flow-run
    would be used by the Agent
    The task family that the Fargate Agent uses should be different from the FargateTaskEnvironment task family
    Carlo

    Carlo

    1 year ago
    I'm not setting a command explicitly in either the flow fargate task env or the agent
    s

    Spencer

    1 year ago
    That's correct
    Carlo

    Carlo

    1 year ago
    I do have family set on the flow fargate task env, but not setting one on the agent
    s

    Spencer

    1 year ago
    So, the Agent will create an ECS Task Definition which when executed will create another Task Definition to run the flow. My intuition is that these are referencing the same definition for some reason.
    Carlo

    Carlo

    1 year ago
    oh, I think it might be because I have taskDefinition and family set to the same value in the fargate task env
    let me change this
    s

    Spencer

    1 year ago
    Well, in the FargateTaskEnvironment, the
    taskDefinition
    and
    family
    are typically the same. I think those values may be the same as the flow-run
    taskDefinition
    that the Agent is creating. You can look in the AWS ECS console to see the existing
    taskDefinition
    Carlo

    Carlo

    1 year ago
    This solved it, different errors now, but I think I can fix these
    s

    Spencer

    1 year ago
    Oh ok. It seems that there could be a really odd scenario where the
    taskDefinition
    here is named the same as a flow's flow-run
    taskDefinition
    would be by the Agent. This probably won't happen if you're not using
    enable_task_revisions
    since you'll get a fairly unique name, otherwise you could have a flow has a name that would collide. ( https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/fargate/agent.py#L478-L487 )
    Carlo

    Carlo

    1 year ago
    I will do some more research and come back
    I think it's because I had
    flow.name
    fargateTaskEnvironment.family
    fargateTaskEnvironment.taskDefinition
    all set to the same value
    I was able to get my flow working. Thanks for the help, it really unblocked me.