Mohan kancherla
03/21/2022, 7:56 PMAn error occurred (InvalidParameterException) when calling the RunTask operation: Override for container named flow is not a container in the TaskDefinition.
Kevin Kho
Mohan kancherla
03/21/2022, 7:57 PMKevin Kho
Mohan kancherla
03/21/2022, 8:00 PM{
"family": "fcc_experimental_licenses",
"executionRoleArn": "arn:aws:iam::AWSACCOUNTNUMBER:role/ecsTaskExecutionRole",
"taskRoleArn": "arn:aws:iam::AWSACCOUNTNUMBER:role/ecsTaskRole",
"networkMode": "awsvpc",
"containerDefinitions": [
{
"name": "fcc_experimental_licenses",
"image": "<http://AWSACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest|AWSACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest>",
"essential": true,
"logConfiguration": {
"logDriver": "awslogs",
"secretOptions": null,
"options": {
"awslogs-group": "/ecs/pipelines",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
},
"secrets": [
{
"name": "snowflake_credentials",
"valueFrom": "arn:aws:secretsmanager:us-west-2:AWSACCOUNTNUMBER:secret:snowflake_credentials-20rxHK"
}
]
}
],
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "1024",
"memory": "2048"
}
Mohan kancherla
03/21/2022, 8:01 PMKevin Kho
flow
for Prefect to know to use that to run the flow inKevin Kho
Mohan kancherla
03/21/2022, 8:11 PMflow
. But how can we use already defined name to it?Kevin Kho
flow
to run the flow inMohan kancherla
03/21/2022, 8:15 PMMohan kancherla
03/21/2022, 8:15 PMMohan kancherla
03/21/2022, 9:57 PMKevin Kho
run_task_kwargs
of the agent and these run_task_kwargs
are passed on to the Flow. Similarly, you can define run_task_kwargs
in ECSRunKevin Kho
Mohan kancherla
03/21/2022, 10:28 PMMohan kancherla
03/21/2022, 10:28 PMKevin Kho
Mohan kancherla
03/21/2022, 10:31 PMKevin Kho
execution_role_arn
or task_role_arn
to the ECSRunKevin Kho
Kevin Kho
task_role_arn
Mohan kancherla
03/21/2022, 10:33 PMKevin Kho
Mohan kancherla
03/21/2022, 10:34 PMKevin Kho
Mohan kancherla
03/21/2022, 10:35 PMKevin Kho
Mohan kancherla
03/21/2022, 10:38 PMMohan kancherla
03/21/2022, 11:29 PMKevin Kho
Mohan kancherla
03/21/2022, 11:30 PMKevin Kho
Mohan kancherla
03/21/2022, 11:31 PMKevin Kho
Mohan kancherla
03/21/2022, 11:32 PMKevin Kho
Kevin Kho
Mohan kancherla
03/21/2022, 11:35 PMimport boto3
from prefect import Flow
from prefect.run_configs import ECSRun
ecr = boto3.client("ecr", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")
ecs = boto3.client("ecs", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dumy2")
flow = Flow("fcc-test")
flow.run_config = ECSRun(task_definition_arn="arn:aws:ecs:us-west-2:NUMBER:task-definition/fcc_experimental_licenses:13", env={"number_of_records":"100000"}, labels=["sandbox"], run_task_kwargs=dict(cluster="Data-Engineering", networkConfiguration={
'awsvpcConfiguration': {
'subnets': [
'subnet-008132793b9c944a0'
],
'securityGroups': [
'sg-01f7c984659e4f3c4',
],
'assignPublicIp': 'ENABLED'
}
}))
flow.register(project_name="test")
Kevin Kho
Mohan kancherla
03/21/2022, 11:36 PMMohan kancherla
03/21/2022, 11:36 PMKevin Kho
Execute a Flow Run
Kevin Kho
Register a Flow
because you need an agent also. In your case, you need the ECS AgentMohan kancherla
03/21/2022, 11:41 PMMohan kancherla
03/21/2022, 11:42 PMKevin Kho
Kevin Kho
Mohan kancherla
03/21/2022, 11:44 PMKevin Kho
Mohan kancherla
03/21/2022, 11:47 PMAnna Geller
Kevin Kho
Mohan kancherla
03/22/2022, 5:23 PMKevin Kho
Mohan kancherla
03/22/2022, 5:26 PMMohan kancherla
03/22/2022, 5:27 PMKevin Kho
Mohan kancherla
03/22/2022, 5:27 PMKevin Kho
Mohan kancherla
03/22/2022, 5:52 PMMohan kancherla
03/22/2022, 5:52 PMKevin Kho
prefect execute flow-run
Anna Geller
Mohan kancherla
03/22/2022, 5:55 PM# Python Instance Settings
FROM python:3.8
# Install Dependencies
RUN pip install snowflake-connector-python
RUN pip install snowflake-sqlalchemy
RUN pip install SQLAlchemy
RUN pip install pyarrow==5.0.0
RUN pip install flask
RUN pip install pandas
RUN pip install requests
RUN pip install boto3
RUN pip install bs4
RUN pip install lxml
RUN pip install prefect
RUN pip install prefect[aws]
# Set Environment Variables
ENV TZ = Etc/UTC
ENV PIPELINE_NAME "FCC_EXPERIMENTAL_LICENSES"
# Create Working Directories
WORKDIR /app
COPY main.py /app/main.py
COPY main_config.py /app/main_config.py
# Make Executable
RUN chmod +x /app/main.py
# Call Script
ENTRYPOINT ["python", "/app/main.py"]
Kevin Kho
main.py
contain your flow here?Mohan kancherla
03/22/2022, 5:57 PMMohan kancherla
03/22/2022, 5:58 PMflow
Mohan kancherla
03/22/2022, 5:59 PMimport boto3
from prefect import Flow
from prefect.run_configs import ECSRun
ecr = boto3.client("ecr", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")
ecs = boto3.client("ecs", region_name="us-west-2", aws_access_key_id="dummy", aws_secret_access_key="dummy2")
flow = Flow("fcc-test")
flow.run_config = ECSRun(task_definition_arn="arn:aws:ecs:us-west-2:ACCOUNNUMBER:task-definition/fcc_experimental_licenses:13", env={"number_of_records":"100000"}, labels=["sandbox"], run_task_kwargs=dict(cluster="Data-Engineering", networkConfiguration={
'awsvpcConfiguration': {
'subnets': [
'subnet-008132793b9c944a0'
],
'securityGroups': [
'sg-01f7c984659e4f3c4',
],
'assignPublicIp': 'ENABLED'
}
}))
flow.register(project_name="test")
Kevin Kho
Kevin Kho
Mohan kancherla
03/22/2022, 6:01 PM{
"family": "fcc_experimental_licenses",
"executionRoleArn": "arn:aws:iam::ACCOUNTNUMBER:role/ecsTaskExecutionRole",
"taskRoleArn": "arn:aws:iam::ACCOUNTNUMBER:role/ecsTaskRole",
"networkMode": "awsvpc",
"containerDefinitions": [
{
"name": "flow",
"image": "<http://ACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest|ACCOUNTNUMBER.dkr.ecr.us-west-2.amazonaws.com/fcc_experimental_licenses:latest>",
"essential": true,
"logConfiguration": {
"logDriver": "awslogs",
"secretOptions": null,
"options": {
"awslogs-group": "/ecs/pipelines",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
},
"secrets": [
{
"name": "snowflake_credentials",
"valueFrom": "arn:aws:secretsmanager:us-west-2:ACCOUNTNUMBER:secret:snowflake_credentials-20rxHK"
}
]
}
],
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "1024",
"memory": "2048"
}
Kevin Kho
Mohan kancherla
03/22/2022, 6:04 PMMohan kancherla
03/22/2022, 6:09 PMKevin Kho
Kevin Kho
main.py
should be brought in to a Prefect Flow, and then PRefect will take care of executionMohan kancherla
03/22/2022, 6:18 PMKevin Kho
Mohan kancherla
03/22/2022, 6:34 PMKevin Kho
prefect register …
in their CI/CD. I don’t have specific resources on Github actions but if you search in this channel you may find 1 or 2 posts