Thread
#prefect-community
    m

    Mohan kancherla

    6 months ago
    Hello Everyone, I am very new to prefect and trying to run ECS container through ECS agent setup. I have provided the task definition arn, env variables, cluster name of ECS to the ECSRUN function and register the flow to the project. When I ran the flow, it gave the below error. Can anyone please explain me more about the error and how we can tackle that?
    An error occurred (InvalidParameterException) when calling the RunTask operation: Override for container named flow is not a container in the TaskDefinition.
    Kevin Kho

    Kevin Kho

    6 months ago
    Can I see the task definition?
    m

    Mohan kancherla

    6 months ago
    Do you mean agent task definition or container task definition?
    Kevin Kho

    Kevin Kho

    6 months ago
    Container
    m

    Mohan kancherla

    6 months ago
    {
      "family": "fcc_experimental_licenses",
      "executionRoleArn": "arn:aws:iam::AWSACCOUNTNUMBER:role/ecsTaskExecutionRole",
      "taskRoleArn": "arn:aws:iam::AWSACCOUNTNUMBER:role/ecsTaskRole",
      "networkMode": "awsvpc",
      "containerDefinitions": [
        {
          "name": "fcc_experimental_licenses",
          "image": "<http://AWSACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest|AWSACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest>",
          "essential": true,
          "logConfiguration": {
            "logDriver": "awslogs",
            "secretOptions": null,
            "options": {
              "awslogs-group": "/ecs/pipelines",
              "awslogs-region": "us-west-2",
              "awslogs-stream-prefix": "ecs"
            }
          },
          "secrets": [
            {
              "name": "snowflake_credentials",
              "valueFrom": "arn:aws:secretsmanager:us-west-2:AWSACCOUNTNUMBER:secret:snowflake_credentials-20rxHK"
            }
          ]
        }
      ],
      "requiresCompatibilities": [
        "FARGATE"
      ],
      "cpu": "1024",
      "memory": "2048"
    }
    for security reason, I have not shared the AWSACCOUNTNUMBER..
    Kevin Kho

    Kevin Kho

    6 months ago
    So the container definitions for ECS needs a container named
    flow
    for Prefect to know to use that to run the flow in
    You can find the docstring for that here
    m

    Mohan kancherla

    6 months ago
    Got it, the container name should be
    flow
    . But how can we use already defined name to it?
    Kevin Kho

    Kevin Kho

    6 months ago
    You can’t because Prefect looks for the container named
    flow
    to run the flow in
    m

    Mohan kancherla

    6 months ago
    okay.
    Thank you.
    @Kevin Kho When I submitted the workflow and started the flow run, it is creating different VPC instead of using the same VPC and security groups of ECS agent? How should we set for that for a flow?
    Kevin Kho

    Kevin Kho

    6 months ago
    I think you set it in the
    run_task_kwargs
    of the agent and these
    run_task_kwargs
    are passed on to the Flow. Similarly, you can define
    run_task_kwargs
    in ECSRun
    This is a good example
    m

    Mohan kancherla

    6 months ago
    Thank you, that helped me. I got one more issue of accessing secrets inside the flow containers. The agent started the flow container but it was unable to access the secrets even though, I have provided the secrets in the task definition file.
    Can you please let me know why?
    Kevin Kho

    Kevin Kho

    6 months ago
    Are those Prefect Secrets?
    m

    Mohan kancherla

    6 months ago
    No, those are secrets in AWS secret manager
    Kevin Kho

    Kevin Kho

    6 months ago
    I would say either add the
    execution_role_arn
    or
    task_role_arn
    to the ECSRun
    Have you seen those?
    I think you want a
    task_role_arn
    m

    Mohan kancherla

    6 months ago
    I have both of them in my task definition file.
    Kevin Kho

    Kevin Kho

    6 months ago
    I would say those guys need the permissions to access secret manager. Do they?
    m

    Mohan kancherla

    6 months ago
    I have seen them but I have already specified both of them in the task definition file.
    Kevin Kho

    Kevin Kho

    6 months ago
    I know but you do they have permissions to access secret manager?
    m

    Mohan kancherla

    6 months ago
    Yes, it has..
    Kevin Kho

    Kevin Kho

    6 months ago
    Are you using the AWSSecretManager task?
    m

    Mohan kancherla

    6 months ago
    It's very weird, the task role unable to access secrets manager. Let me check that and get back to you.
    I figured out the issue and resolved it.
    Kevin Kho

    Kevin Kho

    6 months ago
    What was it? I am curious to know
    m

    Mohan kancherla

    6 months ago
    It was typo issue in the secrets manager.
    Kevin Kho

    Kevin Kho

    6 months ago
    Ahh I see
    m

    Mohan kancherla

    6 months ago
    How to set the flow to be in running status if the container is running mode?
    Kevin Kho

    Kevin Kho

    6 months ago
    If you create the Flow from Prefect, they should go together. You should not create it on ECS task side
    m

    Mohan kancherla

    6 months ago
    I have pushed the flow from my local machine..
    Kevin Kho

    Kevin Kho

    6 months ago
    How did you do that?
    So register is not the same as run, you register first and then run from the Prefect UI
    m

    Mohan kancherla

    6 months ago
    I have authenticated prefect in local machine and register the code.
    import boto3
    from prefect import Flow
    from prefect.run_configs import ECSRun
    
    ecr = boto3.client("ecr", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")
    
    ecs = boto3.client("ecs", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dumy2")
    flow = Flow("fcc-test")
    
    
    flow.run_config = ECSRun(task_definition_arn="arn:aws:ecs:us-west-2:NUMBER:task-definition/fcc_experimental_licenses:13", env={"number_of_records":"100000"}, labels=["sandbox"], run_task_kwargs=dict(cluster="Data-Engineering", networkConfiguration={
                'awsvpcConfiguration': {
                    'subnets': [
                        'subnet-008132793b9c944a0'
                    ],
                    'securityGroups': [
                        'sg-01f7c984659e4f3c4',
                    ],
                    'assignPublicIp': 'ENABLED'
                }
            }))
    
    flow.register(project_name="test")
    Kevin Kho

    Kevin Kho

    6 months ago
    Check this secion
    m

    Mohan kancherla

    6 months ago
    I have pushed the code through terminal.
    How can I setup flow in Prefect through UI?
    Kevin Kho

    Kevin Kho

    6 months ago
    It’s a bit below in that page under
    Execute a Flow Run
    Well you need to check the sections after
    Register a Flow
    because you need an agent also. In your case, you need the ECS Agent
    m

    Mohan kancherla

    6 months ago
    Why did it re-submitted the flow run when the container 1 is still running?
    How to stop the rescheduled by Lazarus process ?
    Kevin Kho

    Kevin Kho

    6 months ago
    In the Flow Page there is Settings and then you can turn off Lazarus
    Did this really run twice on the ECS side? can you check? I think the first submission failed
    m

    Mohan kancherla

    6 months ago
    Yeah, it ran twice on ECS side and the first one didn't failed..
    Kevin Kho

    Kevin Kho

    6 months ago
    This is a bit long to explain. Will respond a bit later
    m

    Mohan kancherla

    6 months ago
    Yeah no worries..
    Anna Geller

    Anna Geller

    6 months ago
    chiming in without knowing the full story 🙃 @Mohan kancherla if you are on Prefect Cloud, you can create an Automation to automatically cancel flow run if it didn't move into Running state after a specific time (SLA). This allows you to cancel a flow run that got stuck in a Submitted state due to ECS issues.
    Kevin Kho

    Kevin Kho

    6 months ago
    So there are times when your Flow won’t be able to get the compute like if it’s running on Kubernetes or ECS. The Flow will hang in a submitted state, so the Zombie Killer and Lazarus are the decide it’s dead and then re-submit it. I guess it’s kind of being aggressive in this case. You can turn it off, but on the other hand you will run into problems when it actually doesn’t spin up as well. It won’t be re-submitted. If that’s fine, you can turn off Lazarus
    m

    Mohan kancherla

    6 months ago
    Thank you for information on Lazarus. But, the question I have is: I have container running with flow and scheduled it. It started running in AWS ECS tasks but the flow timeline is not showing in running status, is there any reason for that?
    Kevin Kho

    Kevin Kho

    6 months ago
    This is a run with an agent? It’s stuck in Submitted?
    m

    Mohan kancherla

    6 months ago
    Yes, it is with ECS agent.
    The task is running in AWS ECS but flow is not showing in running status
    Kevin Kho

    Kevin Kho

    6 months ago
    I haven’t seen this before, or it’s pretty rare. Does this always happen to you?
    m

    Mohan kancherla

    6 months ago
    Yes, it never showed in running status for me.
    Kevin Kho

    Kevin Kho

    6 months ago
    What image are you using for the ECS task definition?
    m

    Mohan kancherla

    6 months ago
    I am using task definition arn
    and the task got failed in ECS tasks but the flow is not showing any status of that.
    Kevin Kho

    Kevin Kho

    6 months ago
    I understand that, but it still has an image attached to it right? I am wondering if you extended the base Prefect image? I think the Flow is not starting because the image doesn’t have the command to trigger flow runs:
    prefect execute flow-run
    Anna Geller

    Anna Geller

    6 months ago
    @Mohan kancherla if you could share your Dockerfile, it would help us a lot with troubleshooting your issue
    m

    Mohan kancherla

    6 months ago
    # Python Instance Settings
    FROM python:3.8
    
    
    # Install Dependencies
    RUN pip install snowflake-connector-python
    RUN pip install snowflake-sqlalchemy
    RUN pip install SQLAlchemy
    RUN pip install pyarrow==5.0.0
    RUN pip install flask
    RUN pip install pandas
    RUN pip install requests
    RUN pip install boto3
    RUN pip install bs4
    RUN pip install lxml
    RUN pip install prefect
    RUN pip install prefect[aws]
    
    # Set Environment Variables
    ENV TZ = Etc/UTC
    ENV PIPELINE_NAME "FCC_EXPERIMENTAL_LICENSES"
    
    # Create Working Directories
    WORKDIR /app
    COPY main.py /app/main.py
    COPY main_config.py /app/main_config.py
    
    # Make Executable
    RUN chmod +x /app/main.py
    
    # Call Script
    ENTRYPOINT ["python", "/app/main.py"]
    Kevin Kho

    Kevin Kho

    6 months ago
    Does
    main.py
    contain your flow here?
    m

    Mohan kancherla

    6 months ago
    Nope, I have flow.py file which invokes the container by using task definition arn.
    I have changed container task definition to include name as
    flow
    This is my flow.py file code
    import boto3
    from prefect import Flow
    from prefect.run_configs import ECSRun
    
    ecr = boto3.client("ecr", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")
    
    ecs = boto3.client("ecs", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")
    flow = Flow("fcc-test")
    
    
    flow.run_config = ECSRun(task_definition_arn="arn:aws:ecs:us-west-2:ACCOUNNUMBER:task-definition/fcc_experimental_licenses:13", env={"number_of_records":"100000"}, labels=["sandbox"], run_task_kwargs=dict(cluster="Data-Engineering", networkConfiguration={
                'awsvpcConfiguration': {
                    'subnets': [
                        'subnet-008132793b9c944a0'
                    ],
                    'securityGroups': [
                        'sg-01f7c984659e4f3c4',
                    ],
                    'assignPublicIp': 'ENABLED'
                }
            }))
    
    flow.register(project_name="test")
    Kevin Kho

    Kevin Kho

    6 months ago
    I think the issue here is your Flow does nothing? It doesn’t contain any tasks
    Are you trying to just run an ARN with the Prefect Flow?
    m

    Mohan kancherla

    6 months ago
    This is my task definition file:
    {
      "family": "fcc_experimental_licenses",
      "executionRoleArn": "arn:aws:iam::ACCOUNTNUMBER:role/ecsTaskExecutionRole",
      "taskRoleArn": "arn:aws:iam::ACCOUNTNUMBER:role/ecsTaskRole",
      "networkMode": "awsvpc",
      "containerDefinitions": [
        {
          "name": "flow",
          "image": "<http://ACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest|ACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest>",
          "essential": true,
          "logConfiguration": {
            "logDriver": "awslogs",
            "secretOptions": null,
            "options": {
              "awslogs-group": "/ecs/pipelines",
              "awslogs-region": "us-west-2",
              "awslogs-stream-prefix": "ecs"
            }
          },
          "secrets": [
            {
              "name": "snowflake_credentials",
              "valueFrom": "arn:aws:secretsmanager:us-west-2:ACCOUNTNUMBER:secret:snowflake_credentials-20rxHK"
            }
          ]
        }
      ],
      "requiresCompatibilities": [
        "FARGATE"
      ],
      "cpu": "1024",
      "memory": "2048"
    }
    Kevin Kho

    Kevin Kho

    6 months ago
    No no. This is the ECS task, but a Prefect Flow needs to contain Prefect tasks . Your flow is empty so it does nothing
    m

    Mohan kancherla

    6 months ago
    My requirement is -> Prefect agent should invoke my container and run it. I have setup the container arn as flow.
    The task is to run the container and keep providing the status of the container running.
    Kevin Kho

    Kevin Kho

    6 months ago
    I think there is some confusion to what Prefect does for you.1. If you write your code in Prefect and construct a Flow with Tasks and then use the ECS Agent + ECS RunConfig, it will spin up a container to run the Flow and then execute it. This is the current set-up that you have. You registered a Flow with Prefect Cloud. When a run is triggered, the Agent submits an ECS Task using that task-definition-arn to spin up a container and then runs your Flow inside that container. The problem is that the flow is empty so it does nothing. I actually think your ENTRYPOINT might still run though, so maybe you can just have a Flow with dummy tasks and then this might work, but the Flow is more of a distraction irrelevant to running the container. 2. If you want to just invoke a container on ECS and run it, I would suggest that you create a separate Flow that invokes the ECS task as a Prefect task by using the boto3 client. Revert your task definition to the old one without the flow container. Create a new Prefect Flow that hits ECS to spin that up
    Both have some awkwardness to it because Prefect is meant to contain your logic also and deploy the logic in the execution environment. Ideally, the content of
    main.py
    should be brought in to a Prefect Flow, and then PRefect will take care of execution
    m

    Mohan kancherla

    6 months ago
    Can you explain little bit more on point 2 or point me if any documentation?
    Kevin Kho

    Kevin Kho

    6 months ago
    I think this may help
    m

    Mohan kancherla

    6 months ago
    I have saved the ecsruntaskusage as per my example provided in the above link. How should I push that to prefect cloud using github CI/CD actions ?
    Kevin Kho

    Kevin Kho

    6 months ago
    You would register that Flow and then maybe run it on a schedule to bring it to Prefect Cloud. You shouldn’t need CI/CD unless you change it a lot, but what some people do here is just register with the CLI
    prefect register …
    in their CI/CD. I don’t have specific resources on Github actions but if you search in this channel you may find 1 or 2 posts