https://prefect.io
Join Slack
Hello Everyone, I am very new to prefect and trying to run ECS container through ECS agent setup. I ...
# ask-community
m
Hello Everyone, I am very new to prefect and trying to run ECS container through ECS agent setup. I have provided the task definition arn, env variables, cluster name of ECS to the ECSRUN function and register the flow to the project. When I ran the flow, it gave the below error. Can anyone please explain me more about the error and how we can tackle that?
Copy code
An error occurred (InvalidParameterException) when calling the RunTask operation: Override for container named flow is not a container in the TaskDefinition.
k
Can I see the task definition?
m
Do you mean agent task definition or container task definition?
k
Container
m
Copy code
{
  "family": "fcc_experimental_licenses",
  "executionRoleArn": "arn:aws:iam::AWSACCOUNTNUMBER:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::AWSACCOUNTNUMBER:role/ecsTaskRole",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "name": "fcc_experimental_licenses",
      "image": "<http://AWSACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest|AWSACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest>",
      "essential": true,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/ecs/pipelines",
          "awslogs-region": "us-west-2",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "secrets": [
        {
          "name": "snowflake_credentials",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:AWSACCOUNTNUMBER:secret:snowflake_credentials-20rxHK"
        }
      ]
    }
  ],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "1024",
  "memory": "2048"
}
for security reason, I have not shared the AWSACCOUNTNUMBER..
k
So the container definitions for ECS needs a container named 
flow
for Prefect to know to use that to run the flow in
You can find the docstring for that here
m
Got it, the container name should be 
flow
. But how can we use already defined name to it?
k
You can’t because Prefect looks for the container named 
flow
to run the flow in
m
okay.
Thank you.
@Kevin Kho When I submitted the workflow and started the flow run, it is creating different VPC instead of using the same VPC and security groups of ECS agent? How should we set for that for a flow?
k
I think you set it in the 
run_task_kwargs
of the agent and these 
run_task_kwargs
are passed on to the Flow. Similarly, you can define 
run_task_kwargs
in ECSRun
This is a good example
m
Thank you, that helped me. I got one more issue of accessing secrets inside the flow containers. The agent started the flow container but it was unable to access the secrets even though, I have provided the secrets in the task definition file.
Can you please let me know why?
k
Are those Prefect Secrets?
m
No, those are secrets in AWS secret manager
k
I would say either add the 
execution_role_arn
or 
task_role_arn
to the ECSRun
Have you seen those?
I think you want a 
task_role_arn
m
I have both of them in my task definition file.
k
I would say those guys need the permissions to access secret manager. Do they?
m
I have seen them but I have already specified both of them in the task definition file.
k
I know but you do they have permissions to access secret manager?
m
Yes, it has..
k
Are you using the AWSSecretManager task?
m
It's very weird, the task role unable to access secrets manager. Let me check that and get back to you.
I figured out the issue and resolved it.
k
What was it? I am curious to know
m
It was typo issue in the secrets manager.
k
Ahh I see
m
How to set the flow to be in running status if the container is running mode?
k
If you create the Flow from Prefect, they should go together. You should not create it on ECS task side
m
I have pushed the flow from my local machine..
k
How did you do that?
So register is not the same as run, you register first and then run from the Prefect UI
m
I have authenticated prefect in local machine and register the code.
Copy code
import boto3
from prefect import Flow
from prefect.run_configs import ECSRun

ecr = boto3.client("ecr", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")

ecs = boto3.client("ecs", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dumy2")
flow = Flow("fcc-test")


flow.run_config = ECSRun(task_definition_arn="arn:aws:ecs:us-west-2:NUMBER:task-definition/fcc_experimental_licenses:13", env={"number_of_records":"100000"}, labels=["sandbox"], run_task_kwargs=dict(cluster="Data-Engineering", networkConfiguration={
            'awsvpcConfiguration': {
                'subnets': [
                    'subnet-008132793b9c944a0'
                ],
                'securityGroups': [
                    'sg-01f7c984659e4f3c4',
                ],
                'assignPublicIp': 'ENABLED'
            }
        }))

flow.register(project_name="test")
k
Check this secion
m
I have pushed the code through terminal.
How can I setup flow in Prefect through UI?
k
It’s a bit below in that page under 
Execute a Flow Run
Well you need to check the sections after 
Register a Flow
because you need an agent also. In your case, you need the ECS Agent
m
Why did it re-submitted the flow run when the container 1 is still running?
How to stop the rescheduled by Lazarus process ?
k
In the Flow Page there is Settings and then you can turn off Lazarus
Did this really run twice on the ECS side? can you check? I think the first submission failed
m
Yeah, it ran twice on ECS side and the first one didn't failed..
k
This is a bit long to explain. Will respond a bit later
m
Yeah no worries..
a
chiming in without knowing the full story 🙃 @Mohan kancherla if you are on Prefect Cloud, you can create an Automation to automatically cancel flow run if it didn't move into Running state after a specific time (SLA). This allows you to cancel a flow run that got stuck in a Submitted state due to ECS issues.
k
So there are times when your Flow won’t be able to get the compute like if it’s running on Kubernetes or ECS. The Flow will hang in a submitted state, so the Zombie Killer and Lazarus are the decide it’s dead and then re-submit it. I guess it’s kind of being aggressive in this case. You can turn it off, but on the other hand you will run into problems when it actually doesn’t spin up as well. It won’t be re-submitted. If that’s fine, you can turn off Lazarus
m
Thank you for information on Lazarus. But, the question I have is: I have container running with flow and scheduled it. It started running in AWS ECS tasks but the flow timeline is not showing in running status, is there any reason for that?
k
This is a run with an agent? It’s stuck in Submitted?
m
Yes, it is with ECS agent.
The task is running in AWS ECS but flow is not showing in running status
k
I haven’t seen this before, or it’s pretty rare. Does this always happen to you?
m
Yes, it never showed in running status for me.
k
What image are you using for the ECS task definition?
m
I am using task definition arn
and the task got failed in ECS tasks but the flow is not showing any status of that.
k
I understand that, but it still has an image attached to it right? I am wondering if you extended the base Prefect image? I think the Flow is not starting because the image doesn’t have the command to trigger flow runs: 
prefect execute flow-run
upvote 1
a
@Mohan kancherla if you could share your Dockerfile, it would help us a lot with troubleshooting your issue
m
Copy code
# Python Instance Settings
FROM python:3.8


# Install Dependencies
RUN pip install snowflake-connector-python
RUN pip install snowflake-sqlalchemy
RUN pip install SQLAlchemy
RUN pip install pyarrow==5.0.0
RUN pip install flask
RUN pip install pandas
RUN pip install requests
RUN pip install boto3
RUN pip install bs4
RUN pip install lxml
RUN pip install prefect
RUN pip install prefect[aws]

# Set Environment Variables
ENV TZ = Etc/UTC
ENV PIPELINE_NAME "FCC_EXPERIMENTAL_LICENSES"

# Create Working Directories
WORKDIR /app
COPY main.py /app/main.py
COPY main_config.py /app/main_config.py

# Make Executable
RUN chmod +x /app/main.py

# Call Script
ENTRYPOINT ["python", "/app/main.py"]
k
Does 
main.py
contain your flow here?
m
Nope, I have flow.py file which invokes the container by using task definition arn.
I have changed container task definition to include name as 
flow
This is my flow.py file code
Copy code
import boto3
from prefect import Flow
from prefect.run_configs import ECSRun

ecr = boto3.client("ecr", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")

ecs = boto3.client("ecs", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")
flow = Flow("fcc-test")


flow.run_config = ECSRun(task_definition_arn="arn:aws:ecs:us-west-2:ACCOUNNUMBER:task-definition/fcc_experimental_licenses:13", env={"number_of_records":"100000"}, labels=["sandbox"], run_task_kwargs=dict(cluster="Data-Engineering", networkConfiguration={
            'awsvpcConfiguration': {
                'subnets': [
                    'subnet-008132793b9c944a0'
                ],
                'securityGroups': [
                    'sg-01f7c984659e4f3c4',
                ],
                'assignPublicIp': 'ENABLED'
            }
        }))

flow.register(project_name="test")
k
I think the issue here is your Flow does nothing? It doesn’t contain any tasks
Are you trying to just run an ARN with the Prefect Flow?
m
This is my task definition file:
Copy code
{
  "family": "fcc_experimental_licenses",
  "executionRoleArn": "arn:aws:iam::ACCOUNTNUMBER:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::ACCOUNTNUMBER:role/ecsTaskRole",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "name": "flow",
      "image": "<http://ACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest|ACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest>",
      "essential": true,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/ecs/pipelines",
          "awslogs-region": "us-west-2",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "secrets": [
        {
          "name": "snowflake_credentials",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:ACCOUNTNUMBER:secret:snowflake_credentials-20rxHK"
        }
      ]
    }
  ],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "1024",
  "memory": "2048"
}
k
No no. This is the ECS task, but a Prefect Flow needs to contain Prefect tasks . Your flow is empty so it does nothing
m
My requirement is -> Prefect agent should invoke my container and run it. I have setup the container arn as flow.
The task is to run the container and keep providing the status of the container running.
k
I think there is some confusion to what Prefect does for you. 1. If you write your code in Prefect and construct a Flow with Tasks and then use the ECS Agent + ECS RunConfig, it will spin up a container to run the Flow and then execute it. This is the current set-up that you have. You registered a Flow with Prefect Cloud. When a run is triggered, the Agent submits an ECS Task using that task-definition-arn to spin up a container and then runs your Flow inside that container. The problem is that the flow is empty so it does nothing. I actually think your ENTRYPOINT might still run though, so maybe you can just have a Flow with dummy tasks and then this might work, but the Flow is more of a distraction irrelevant to running the container. 2. If you want to just invoke a container on ECS and run it, I would suggest that you create a separate Flow that invokes the ECS task as a Prefect task by using the boto3 client. Revert your task definition to the old one without the flow container. Create a new Prefect Flow that hits ECS to spin that up
Both have some awkwardness to it because Prefect is meant to contain your logic also and deploy the logic in the execution environment. Ideally, the content of 
main.py
should be brought in to a Prefect Flow, and then PRefect will take care of execution
m
Can you explain little bit more on point 2 or point me if any documentation?
k
I think this may help
m
I have saved the ecsruntaskusage as per my example provided in the above link. How should I push that to prefect cloud using github CI/CD actions ?
k
You would register that Flow and then maybe run it on a schedule to bring it to Prefect Cloud. You shouldn’t need CI/CD unless you change it a lot, but what some people do here is just register with the CLI 
prefect register …
in their CI/CD. I don’t have specific resources on Github actions but if you search in this channel you may find 1 or 2 posts
13 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by