https://prefect.io logo
Channels
Powered by
#prefect-community
Title
# prefect-community
m

Mohan kancherla

03/21/2022, 7:56 PM
Hello Everyone, I am very new to prefect and trying to run ECS container through ECS agent setup. I have provided the task definition arn, env variables, cluster name of ECS to the ECSRUN function and register the flow to the project. When I ran the flow, it gave the below error. Can anyone please explain me more about the error and how we can tackle that?
Copy code
An error occurred (InvalidParameterException) when calling the RunTask operation: Override for container named flow is not a container in the TaskDefinition.
k

Kevin Kho

03/21/2022, 7:57 PM
Can I see the task definition?
m

Mohan kancherla

03/21/2022, 7:57 PM
Do you mean agent task definition or container task definition?
k

Kevin Kho

03/21/2022, 7:58 PM
Container
m

Mohan kancherla

03/21/2022, 8:00 PM
Copy code
{
  "family": "fcc_experimental_licenses",
  "executionRoleArn": "arn:aws:iam::AWSACCOUNTNUMBER:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::AWSACCOUNTNUMBER:role/ecsTaskRole",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "name": "fcc_experimental_licenses",
      "image": "<http://AWSACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest|AWSACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest>",
      "essential": true,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/ecs/pipelines",
          "awslogs-region": "us-west-2",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "secrets": [
        {
          "name": "snowflake_credentials",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:AWSACCOUNTNUMBER:secret:snowflake_credentials-20rxHK"
        }
      ]
    }
  ],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "1024",
  "memory": "2048"
}
for security reason, I have not shared the AWSACCOUNTNUMBER..
k

Kevin Kho

03/21/2022, 8:03 PM
So the container definitions for ECS needs a container named 
flow
for Prefect to know to use that to run the flow in
You can find the docstring for that here
m

Mohan kancherla

03/21/2022, 8:11 PM
Got it, the container name should be 
flow
. But how can we use already defined name to it?
k

Kevin Kho

03/21/2022, 8:14 PM
You can’t because Prefect looks for the container named 
flow
to run the flow in
m

Mohan kancherla

03/21/2022, 8:15 PM
okay.
Thank you.
@Kevin Kho When I submitted the workflow and started the flow run, it is creating different VPC instead of using the same VPC and security groups of ECS agent? How should we set for that for a flow?
k

Kevin Kho

03/21/2022, 9:59 PM
I think you set it in the 
run_task_kwargs
of the agent and these 
run_task_kwargs
are passed on to the Flow. Similarly, you can define 
run_task_kwargs
in ECSRun
This is a good example
m

Mohan kancherla

03/21/2022, 10:28 PM
Thank you, that helped me. I got one more issue of accessing secrets inside the flow containers. The agent started the flow container but it was unable to access the secrets even though, I have provided the secrets in the task definition file.
Can you please let me know why?
k

Kevin Kho

03/21/2022, 10:30 PM
Are those Prefect Secrets?
m

Mohan kancherla

03/21/2022, 10:31 PM
No, those are secrets in AWS secret manager
k

Kevin Kho

03/21/2022, 10:32 PM
I would say either add the 
execution_role_arn
or 
task_role_arn
to the ECSRun
Have you seen those?
I think you want a 
task_role_arn
m

Mohan kancherla

03/21/2022, 10:33 PM
I have both of them in my task definition file.
k

Kevin Kho

03/21/2022, 10:34 PM
I would say those guys need the permissions to access secret manager. Do they?
m

Mohan kancherla

03/21/2022, 10:34 PM
I have seen them but I have already specified both of them in the task definition file.
k

Kevin Kho

03/21/2022, 10:35 PM
I know but you do they have permissions to access secret manager?
m

Mohan kancherla

03/21/2022, 10:35 PM
Yes, it has..
k

Kevin Kho

03/21/2022, 10:35 PM
Are you using the AWSSecretManager task?
m

Mohan kancherla

03/21/2022, 10:38 PM
It's very weird, the task role unable to access secrets manager. Let me check that and get back to you.
I figured out the issue and resolved it.
k

Kevin Kho

03/21/2022, 11:30 PM
What was it? I am curious to know
m

Mohan kancherla

03/21/2022, 11:30 PM
It was typo issue in the secrets manager.
k

Kevin Kho

03/21/2022, 11:30 PM
Ahh I see
m

Mohan kancherla

03/21/2022, 11:31 PM
How to set the flow to be in running status if the container is running mode?
k

Kevin Kho

03/21/2022, 11:31 PM
If you create the Flow from Prefect, they should go together. You should not create it on ECS task side
m

Mohan kancherla

03/21/2022, 11:32 PM
I have pushed the flow from my local machine..
k

Kevin Kho

03/21/2022, 11:32 PM
How did you do that?
So register is not the same as run, you register first and then run from the Prefect UI
m

Mohan kancherla

03/21/2022, 11:35 PM
I have authenticated prefect in local machine and register the code.
Copy code
import boto3
from prefect import Flow
from prefect.run_configs import ECSRun

ecr = boto3.client("ecr", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")

ecs = boto3.client("ecs", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dumy2")
flow = Flow("fcc-test")


flow.run_config = ECSRun(task_definition_arn="arn:aws:ecs:us-west-2:NUMBER:task-definition/fcc_experimental_licenses:13", env={"number_of_records":"100000"}, labels=["sandbox"], run_task_kwargs=dict(cluster="Data-Engineering", networkConfiguration={
            'awsvpcConfiguration': {
                'subnets': [
                    'subnet-008132793b9c944a0'
                ],
                'securityGroups': [
                    'sg-01f7c984659e4f3c4',
                ],
                'assignPublicIp': 'ENABLED'
            }
        }))

flow.register(project_name="test")
k

Kevin Kho

03/21/2022, 11:35 PM
Check this secion
m

Mohan kancherla

03/21/2022, 11:36 PM
I have pushed the code through terminal.
How can I setup flow in Prefect through UI?
k

Kevin Kho

03/21/2022, 11:39 PM
It’s a bit below in that page under 
Execute a Flow Run
Well you need to check the sections after 
Register a Flow
because you need an agent also. In your case, you need the ECS Agent
m

Mohan kancherla

03/21/2022, 11:41 PM
Why did it re-submitted the flow run when the container 1 is still running?
How to stop the rescheduled by Lazarus process ?
k

Kevin Kho

03/21/2022, 11:43 PM
In the Flow Page there is Settings and then you can turn off Lazarus
Did this really run twice on the ECS side? can you check? I think the first submission failed
m

Mohan kancherla

03/21/2022, 11:44 PM
Yeah, it ran twice on ECS side and the first one didn't failed..
k

Kevin Kho

03/21/2022, 11:45 PM
This is a bit long to explain. Will respond a bit later
m

Mohan kancherla

03/21/2022, 11:47 PM
Yeah no worries..
a

Anna Geller

03/22/2022, 12:38 AM
chiming in without knowing the full story 🙃 @Mohan kancherla if you are on Prefect Cloud, you can create an Automation to automatically cancel flow run if it didn't move into Running state after a specific time (SLA). This allows you to cancel a flow run that got stuck in a Submitted state due to ECS issues.
k

Kevin Kho

03/22/2022, 3:35 AM
So there are times when your Flow won’t be able to get the compute like if it’s running on Kubernetes or ECS. The Flow will hang in a submitted state, so the Zombie Killer and Lazarus are the decide it’s dead and then re-submit it. I guess it’s kind of being aggressive in this case. You can turn it off, but on the other hand you will run into problems when it actually doesn’t spin up as well. It won’t be re-submitted. If that’s fine, you can turn off Lazarus
m

Mohan kancherla

03/22/2022, 5:23 PM
Thank you for information on Lazarus. But, the question I have is: I have container running with flow and scheduled it. It started running in AWS ECS tasks but the flow timeline is not showing in running status, is there any reason for that?
k

Kevin Kho

03/22/2022, 5:26 PM
This is a run with an agent? It’s stuck in Submitted?
m

Mohan kancherla

03/22/2022, 5:26 PM
Yes, it is with ECS agent.
The task is running in AWS ECS but flow is not showing in running status
k

Kevin Kho

03/22/2022, 5:27 PM
I haven’t seen this before, or it’s pretty rare. Does this always happen to you?
m

Mohan kancherla

03/22/2022, 5:27 PM
Yes, it never showed in running status for me.
k

Kevin Kho

03/22/2022, 5:30 PM
What image are you using for the ECS task definition?
m

Mohan kancherla

03/22/2022, 5:52 PM
I am using task definition arn
and the task got failed in ECS tasks but the flow is not showing any status of that.
k

Kevin Kho

03/22/2022, 5:54 PM
I understand that, but it still has an image attached to it right? I am wondering if you extended the base Prefect image? I think the Flow is not starting because the image doesn’t have the command to trigger flow runs: 
prefect execute flow-run
upvote 1
a

Anna Geller

03/22/2022, 5:55 PM
@Mohan kancherla if you could share your Dockerfile, it would help us a lot with troubleshooting your issue
m

Mohan kancherla

03/22/2022, 5:55 PM
Copy code
# Python Instance Settings
FROM python:3.8


# Install Dependencies
RUN pip install snowflake-connector-python
RUN pip install snowflake-sqlalchemy
RUN pip install SQLAlchemy
RUN pip install pyarrow==5.0.0
RUN pip install flask
RUN pip install pandas
RUN pip install requests
RUN pip install boto3
RUN pip install bs4
RUN pip install lxml
RUN pip install prefect
RUN pip install prefect[aws]

# Set Environment Variables
ENV TZ = Etc/UTC
ENV PIPELINE_NAME "FCC_EXPERIMENTAL_LICENSES"

# Create Working Directories
WORKDIR /app
COPY main.py /app/main.py
COPY main_config.py /app/main_config.py

# Make Executable
RUN chmod +x /app/main.py

# Call Script
ENTRYPOINT ["python", "/app/main.py"]
k

Kevin Kho

03/22/2022, 5:56 PM
Does 
main.py
contain your flow here?
m

Mohan kancherla

03/22/2022, 5:57 PM
Nope, I have flow.py file which invokes the container by using task definition arn.
I have changed container task definition to include name as 
flow
This is my flow.py file code
Copy code
import boto3
from prefect import Flow
from prefect.run_configs import ECSRun

ecr = boto3.client("ecr", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")

ecs = boto3.client("ecs", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")
flow = Flow("fcc-test")


flow.run_config = ECSRun(task_definition_arn="arn:aws:ecs:us-west-2:ACCOUNNUMBER:task-definition/fcc_experimental_licenses:13", env={"number_of_records":"100000"}, labels=["sandbox"], run_task_kwargs=dict(cluster="Data-Engineering", networkConfiguration={
            'awsvpcConfiguration': {
                'subnets': [
                    'subnet-008132793b9c944a0'
                ],
                'securityGroups': [
                    'sg-01f7c984659e4f3c4',
                ],
                'assignPublicIp': 'ENABLED'
            }
        }))

flow.register(project_name="test")
k

Kevin Kho

03/22/2022, 6:00 PM
I think the issue here is your Flow does nothing? It doesn’t contain any tasks
Are you trying to just run an ARN with the Prefect Flow?
m

Mohan kancherla

03/22/2022, 6:01 PM
This is my task definition file:
Copy code
{
  "family": "fcc_experimental_licenses",
  "executionRoleArn": "arn:aws:iam::ACCOUNTNUMBER:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::ACCOUNTNUMBER:role/ecsTaskRole",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "name": "flow",
      "image": "<http://ACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest|ACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest>",
      "essential": true,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/ecs/pipelines",
          "awslogs-region": "us-west-2",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "secrets": [
        {
          "name": "snowflake_credentials",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:ACCOUNTNUMBER:secret:snowflake_credentials-20rxHK"
        }
      ]
    }
  ],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "1024",
  "memory": "2048"
}
k

Kevin Kho

03/22/2022, 6:02 PM
No no. This is the ECS task, but a Prefect Flow needs to contain Prefect tasks . Your flow is empty so it does nothing
m

Mohan kancherla

03/22/2022, 6:04 PM
My requirement is -> Prefect agent should invoke my container and run it. I have setup the container arn as flow.
The task is to run the container and keep providing the status of the container running.
k

Kevin Kho

03/22/2022, 6:10 PM
I think there is some confusion to what Prefect does for you. 1. If you write your code in Prefect and construct a Flow with Tasks and then use the ECS Agent + ECS RunConfig, it will spin up a container to run the Flow and then execute it. This is the current set-up that you have. You registered a Flow with Prefect Cloud. When a run is triggered, the Agent submits an ECS Task using that task-definition-arn to spin up a container and then runs your Flow inside that container. The problem is that the flow is empty so it does nothing. I actually think your ENTRYPOINT might still run though, so maybe you can just have a Flow with dummy tasks and then this might work, but the Flow is more of a distraction irrelevant to running the container. 2. If you want to just invoke a container on ECS and run it, I would suggest that you create a separate Flow that invokes the ECS task as a Prefect task by using the boto3 client. Revert your task definition to the old one without the flow container. Create a new Prefect Flow that hits ECS to spin that up
Both have some awkwardness to it because Prefect is meant to contain your logic also and deploy the logic in the execution environment. Ideally, the content of 
main.py
should be brought in to a Prefect Flow, and then PRefect will take care of execution
m

Mohan kancherla

03/22/2022, 6:18 PM
Can you explain little bit more on point 2 or point me if any documentation?
k

Kevin Kho

03/22/2022, 6:19 PM
I think this may help
m

Mohan kancherla

03/22/2022, 6:34 PM
I have saved the ecsruntaskusage as per my example provided in the above link. How should I push that to prefect cloud using github CI/CD actions ?
k

Kevin Kho

03/22/2022, 6:44 PM
You would register that Flow and then maybe run it on a schedule to bring it to Prefect Cloud. You shouldn’t need CI/CD unless you change it a lot, but what some people do here is just register with the CLI 
prefect register …
in their CI/CD. I don’t have specific resources on Github actions but if you search in this channel you may find 1 or 2 posts
4 Views
Powered by