Mohan kancherla
03/21/2022, 7:56 PMAn error occurred (InvalidParameterException) when calling the RunTask operation: Override for container named flow is not a container in the TaskDefinition.
Kevin Kho
03/21/2022, 7:57 PMMohan kancherla
03/21/2022, 7:57 PMKevin Kho
03/21/2022, 7:58 PMMohan kancherla
03/21/2022, 8:00 PM{
"family": "fcc_experimental_licenses",
"executionRoleArn": "arn:aws:iam::AWSACCOUNTNUMBER:role/ecsTaskExecutionRole",
"taskRoleArn": "arn:aws:iam::AWSACCOUNTNUMBER:role/ecsTaskRole",
"networkMode": "awsvpc",
"containerDefinitions": [
{
"name": "fcc_experimental_licenses",
"image": "<http://AWSACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest|AWSACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest>",
"essential": true,
"logConfiguration": {
"logDriver": "awslogs",
"secretOptions": null,
"options": {
"awslogs-group": "/ecs/pipelines",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
},
"secrets": [
{
"name": "snowflake_credentials",
"valueFrom": "arn:aws:secretsmanager:us-west-2:AWSACCOUNTNUMBER:secret:snowflake_credentials-20rxHK"
}
]
}
],
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "1024",
"memory": "2048"
}
Kevin Kho
03/21/2022, 8:03 PMflow
for Prefect to know to use that to run the flow inMohan kancherla
03/21/2022, 8:11 PMflow
. But how can we use already defined name to it?Kevin Kho
03/21/2022, 8:14 PMflow
to run the flow inMohan kancherla
03/21/2022, 8:15 PMKevin Kho
03/21/2022, 9:59 PMrun_task_kwargs
of the agent and these run_task_kwargs
are passed on to the Flow. Similarly, you can define run_task_kwargs
in ECSRunMohan kancherla
03/21/2022, 10:28 PMKevin Kho
03/21/2022, 10:30 PMMohan kancherla
03/21/2022, 10:31 PMKevin Kho
03/21/2022, 10:32 PMexecution_role_arn
or task_role_arn
to the ECSRuntask_role_arn
Mohan kancherla
03/21/2022, 10:33 PMKevin Kho
03/21/2022, 10:34 PMMohan kancherla
03/21/2022, 10:34 PMKevin Kho
03/21/2022, 10:35 PMMohan kancherla
03/21/2022, 10:35 PMKevin Kho
03/21/2022, 10:35 PMMohan kancherla
03/21/2022, 10:38 PMKevin Kho
03/21/2022, 11:30 PMMohan kancherla
03/21/2022, 11:30 PMKevin Kho
03/21/2022, 11:30 PMMohan kancherla
03/21/2022, 11:31 PMKevin Kho
03/21/2022, 11:31 PMMohan kancherla
03/21/2022, 11:32 PMKevin Kho
03/21/2022, 11:32 PMMohan kancherla
03/21/2022, 11:35 PMimport boto3
from prefect import Flow
from prefect.run_configs import ECSRun
ecr = boto3.client("ecr", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")
ecs = boto3.client("ecs", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dumy2")
flow = Flow("fcc-test")
flow.run_config = ECSRun(task_definition_arn="arn:aws:ecs:us-west-2:NUMBER:task-definition/fcc_experimental_licenses:13", env={"number_of_records":"100000"}, labels=["sandbox"], run_task_kwargs=dict(cluster="Data-Engineering", networkConfiguration={
'awsvpcConfiguration': {
'subnets': [
'subnet-008132793b9c944a0'
],
'securityGroups': [
'sg-01f7c984659e4f3c4',
],
'assignPublicIp': 'ENABLED'
}
}))
flow.register(project_name="test")
Kevin Kho
03/21/2022, 11:35 PMMohan kancherla
03/21/2022, 11:36 PMKevin Kho
03/21/2022, 11:39 PMExecute a Flow Run
Register a Flow
because you need an agent also. In your case, you need the ECS AgentMohan kancherla
03/21/2022, 11:41 PMKevin Kho
03/21/2022, 11:43 PMMohan kancherla
03/21/2022, 11:44 PMKevin Kho
03/21/2022, 11:45 PMMohan kancherla
03/21/2022, 11:47 PMAnna Geller
03/22/2022, 12:38 AMKevin Kho
03/22/2022, 3:35 AMMohan kancherla
03/22/2022, 5:23 PMKevin Kho
03/22/2022, 5:26 PMMohan kancherla
03/22/2022, 5:26 PMKevin Kho
03/22/2022, 5:27 PMMohan kancherla
03/22/2022, 5:27 PMKevin Kho
03/22/2022, 5:30 PMMohan kancherla
03/22/2022, 5:52 PMKevin Kho
03/22/2022, 5:54 PMprefect execute flow-run
Anna Geller
03/22/2022, 5:55 PMMohan kancherla
03/22/2022, 5:55 PM# Python Instance Settings
FROM python:3.8
# Install Dependencies
RUN pip install snowflake-connector-python
RUN pip install snowflake-sqlalchemy
RUN pip install SQLAlchemy
RUN pip install pyarrow==5.0.0
RUN pip install flask
RUN pip install pandas
RUN pip install requests
RUN pip install boto3
RUN pip install bs4
RUN pip install lxml
RUN pip install prefect
RUN pip install prefect[aws]
# Set Environment Variables
ENV TZ = Etc/UTC
ENV PIPELINE_NAME "FCC_EXPERIMENTAL_LICENSES"
# Create Working Directories
WORKDIR /app
COPY main.py /app/main.py
COPY main_config.py /app/main_config.py
# Make Executable
RUN chmod +x /app/main.py
# Call Script
ENTRYPOINT ["python", "/app/main.py"]
Kevin Kho
03/22/2022, 5:56 PMmain.py
contain your flow here?Mohan kancherla
03/22/2022, 5:57 PMflow
import boto3
from prefect import Flow
from prefect.run_configs import ECSRun
ecr = boto3.client("ecr", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")
ecs = boto3.client("ecs", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")
flow = Flow("fcc-test")
flow.run_config = ECSRun(task_definition_arn="arn:aws:ecs:us-west-2:ACCOUNNUMBER:task-definition/fcc_experimental_licenses:13", env={"number_of_records":"100000"}, labels=["sandbox"], run_task_kwargs=dict(cluster="Data-Engineering", networkConfiguration={
'awsvpcConfiguration': {
'subnets': [
'subnet-008132793b9c944a0'
],
'securityGroups': [
'sg-01f7c984659e4f3c4',
],
'assignPublicIp': 'ENABLED'
}
}))
flow.register(project_name="test")
Kevin Kho
03/22/2022, 6:00 PMMohan kancherla
03/22/2022, 6:01 PM{
"family": "fcc_experimental_licenses",
"executionRoleArn": "arn:aws:iam::ACCOUNTNUMBER:role/ecsTaskExecutionRole",
"taskRoleArn": "arn:aws:iam::ACCOUNTNUMBER:role/ecsTaskRole",
"networkMode": "awsvpc",
"containerDefinitions": [
{
"name": "flow",
"image": "<http://ACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest|ACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest>",
"essential": true,
"logConfiguration": {
"logDriver": "awslogs",
"secretOptions": null,
"options": {
"awslogs-group": "/ecs/pipelines",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
},
"secrets": [
{
"name": "snowflake_credentials",
"valueFrom": "arn:aws:secretsmanager:us-west-2:ACCOUNTNUMBER:secret:snowflake_credentials-20rxHK"
}
]
}
],
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "1024",
"memory": "2048"
}
Kevin Kho
03/22/2022, 6:02 PMMohan kancherla
03/22/2022, 6:04 PMKevin Kho
03/22/2022, 6:10 PMmain.py
should be brought in to a Prefect Flow, and then PRefect will take care of executionMohan kancherla
03/22/2022, 6:18 PMKevin Kho
03/22/2022, 6:19 PMMohan kancherla
03/22/2022, 6:34 PMKevin Kho
03/22/2022, 6:44 PMprefect register …
in their CI/CD. I don’t have specific resources on Github actions but if you search in this channel you may find 1 or 2 posts