Gagan Singh Saluja
11/30/2021, 1:16 PMAnna Geller
Gagan Singh Saluja
11/30/2021, 1:30 PMGagan Singh Saluja
11/30/2021, 1:31 PMGagan Singh Saluja
11/30/2021, 1:31 PMGagan Singh Saluja
11/30/2021, 1:44 PMAnna Geller
ECSRun(labels=["prod"])
Michael Moscater
01/26/2022, 10:21 PMAnna Geller
"environment": [
{
"name": "PREFECT__CLOUD__AGENT__LABELS",
"value": "['your_label']"
},
{
"name": "PREFECT__CLOUD__AGENT__LEVEL",
"value": "INFO"
},
{
"name": "PREFECT__CLOUD__API",
"value": "paste_your_Server_IP_here"
}
Anna Geller
Michael Moscater
01/26/2022, 10:26 PMMichael Moscater
01/26/2022, 10:44 PMrequests.exceptions.ConnectTimeout: HTTPConnectionPool(host='x.x.x.x', port=4200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f66a952e430>, 'Connection to x.x.x.x timed out. (connect timeout=15)'))
Michael Moscater
01/26/2022, 10:45 PMMichael Moscater
01/26/2022, 10:50 PMMichael Moscater
01/26/2022, 10:50 PMAnna Geller
Michael Moscater
01/27/2022, 3:01 PMMichael Moscater
01/27/2022, 3:02 PMMichael Moscater
01/27/2022, 3:04 PMRUN_CONFIG = ECSRun(
labels=["dev"],
task_definition_path="<s3://bucket/folder/flow_task_definition.yaml>",
run_task_kwargs=dict(cluster="prefectEcsCluster", launchType="FARGATE",)
Michael Moscater
01/27/2022, 3:07 PMfamily: prefectFlow
requiresCompatibilities:
- FARGATE
networkMode: awsvpc
cpu: 1024
memory: 2048
taskRoleArn: arn:aws:iam::XXXX:role/ECSTaskS3ECRRole
executionRoleArn: arn:aws:iam::XXXX:role/ecsTaskExecutionRole
containerDefinitions:
- name: flow
image: "<http://XXXX.dkr.ecr.us-east-2.amazonaws.com/ecsflows:latest|XXXX.dkr.ecr.us-east-2.amazonaws.com/ecsflows:latest>"
essential: true
environment:
- name: AWS_RETRY_MODE
value: "adaptive"
- name: AWS_MAX_ATTEMPTS
value: "10"
- name: PREFECT__BACKEND
value: "server"
- name: PREFECT__CLOUD__API
value: "<http://x.x.x.x:4200>"
- name: PREFECT__CLOUD__AGENT__LABELS
value: "['dev']"
- name: PREFECT__CLOUD__AGENT__LEVEL
value: "INFO"
logConfiguration:
logDriver: awslogs
options:
awslogs-group: "/ecs/prefectEcsAgent"
awslogs-region: "us-east-2"
awslogs-stream-prefix: "ecs"
awslogs-create-group: "true"
Anna Geller
<http://x.x.x.x:4200>
Perhaps you can test it by spinning up a single ECS task with a container that tries to ping this endpoint. You could e.g. send a “hello” GraphQL query and check if you get an answer:
query {
hello
}
I suspect that you either have to:
• enable public IP on both ECS tasks and on your Server instance so that flow run containers communicate with each other via public IP
• or ensure that your ECS tasks with flow run containers get deployed to the same subnet as your EC2 Server instance e.g. by specifying networking configuration on your ECSRun:
from prefect.run_configs import ECSRun
RUN_CONFIG = ECSRun(
labels=["dev"],
run_task_kwargs=dict(
cluster="prefectEcsCluster",
networkConfiguration={
"awsvpcConfiguration": {
"subnets": ["subnet_xxx",],
"securityGroups": ["xxxx",],
"assignPublicIp": "ENABLED", # or 'DISABLED'
}
},
),
)
Michael Moscater
01/27/2022, 3:49 PMMichael Moscater
01/27/2022, 8:34 PMAnna Geller
Michael Moscater
01/27/2022, 8:49 PM