Need help deploying Prefect Orion and Prefect Agent to ECS Fargate Clusters - Prefect 2.7.3 =========================================================================== Hi, I am trying to deploy Prefect Orion and Prefect Agent to ECS Fargate clusters. I've had a successful deployment with Docker-Compose but somehow I've been running around in circles deploying it to Fargate. Here's what I am trying to do - 1. Prefect Orion server - running on a Fargate task 1 with Public IP 2. Prefect Agent 1 - running on Fargate task 2, looks up to the deployed Orion server 3. Prefect Agent 2 - running on Fargate task 3, looks up to the deployed Orion server Before anything moves further, I plan on upgrading future tasks to run as ECS tasks (still need to play around with that, but it's something for later) What I have done so far is - create an ECR image built from custom flows and tasks that we have put in, create - 1. ECS Cluster 2. ECS Service and task for Prefect Orion that uses latest Prefect image from Docker-Hub 3. ECS Service and task for Agent-1 that uses custom image from ECR repo built over Prefect's latest docker image 4. ECS Service and task for Agent-2 that uses custom image from ECR repo built over Prefect's latest docker image I've put in a AWS CloudMap to generate a internal network that allows Prefect's agents to look up for Prefect Orion server using internally resolved DNS. This is running into few issues - 1. Prefect Orion starts but throws an error - Can't connect to Orion API at <CloudMap resolved DNS>/api or <CloudMap resolved DNS>:4200/api or 0.0.0.0:4200/api. The task definition does contain Prefect API listed as an environment variable and PostgreSQL DB fetched from AWS SecretsManager. The web UI is running up but can't seem to connect to the DB or show anything unders /setings. (Attached sreenshot) 2. Prefect agents on both the tasks are not starting up and fail with this error - Resourceinitializationerror: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 5 time(s): failed to fetch secret arn:aws:secretsmanager:_**_** from secrets manager: RequestCanceled: request context canceled caused by: context deadline exceeded. The same secret passed to Prefect Orion Fargate task seem to work just fine.

Hey @Amruth VVKP admittedly I'm not an expert on ECS though you might try running through some of the suggestions mentioned here https://discourse.prefect.io/t/deploy-prefect-in-a-container-on-a-remote-vm/1915/17, There's a lot of overhead that can be difficult to pinpoint through a slack thread so some of those suggestion may help get you in the right direction.

I've managed to resolve issue #2, Prefect agents are starting up just fine. The problem was that assign_public_ip was initially set to False which resulted in failure of fetching secrets from AWS SecretsManager.

Hmm I don't know if outside of the suggestion in the article I sent previously there's anything else i could add, generally speaking I'd say to make sure that you have at least these settings configured, there may be more necessary but these are the ones off the top of my head that I'm pretty sure need to be set • PREFECT_API_URL • PREFECT_ORION_API_HOST • PREFECT_ORION_API_PORT • PREFECT_ORION_DATABASE_CONNECTION_URL • PREFECT_ORION_DATABASE_PASSWORD (if necessary)

I am not sure what I am missing through, this is my terraform task definition

# ECS Task Definition for Prefect's Orchestrator task
resource "aws_ecs_task_definition" "prefect_orchestrator_task" {
  family = var.prefect_orchestrator_name
  cpu    = var.agent_cpu
  memory = var.agent_memory

  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"

  // Execution role allows ECS to create tasks and services
  execution_role_arn = data.aws_iam_role.ecs_task_execution_role.arn

  // Task role allows tasks and services to access other AWS resources
  task_role_arn = data.aws_iam_role.prefect_orchestrator_role.arn

  container_definitions = jsonencode([
    {
      name  = "${var.prefect_orchestrator_name}-task-definition"
      image = "${var.orchestrator_image}" # var.orchestrator_image = prefecthq/prefect:2-latest

      command = ["prefect", "orion", "start", "--host", "0.0.0.0", "--log-level", "WARNING"]

      essential = true

      cpu    = "${var.agent_cpu}"
      memory = "${var.agent_memory}"

      portMappings = [
        {
          name          = var.orchestrator_service_port_name
          containerPort = var.prefect_port # var.prefect_port = 4200
          hostPort      = var.prefect_port
          appProtocol   = "http"
        }
      ]

      environment = [
        {
          name  = "PREFECT_API_URL "
          value = "<http://0.0.0.0>:${var.prefect_port}/api"
        },
        {
          name  = "PREFECT_ORION_API_HOST "
          value = "0.0.0.0"
        },
        {
          name  = "PREFECT_ORION_API_PORT "
          value = tostring("${var.prefect_port}")
        }
      ]
      secrets = [
        {
          name      = "PREFECT_ORION_DATABASE_CONNECTION_URL"
          valueFrom = "${aws_secretsmanager_secret.prefect_postgres_db_connection.arn}"
        }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group         = "${aws_cloudwatch_log_group.prefect_orchestrator_log_group.name}"
          awslogs-region        = "${data.aws_region.current.name}"
          awslogs-stream-prefix = "${var.prefect_orchestrator_name}"
        }
      }
    }

  ])
}

The environment variables from the ECS task looks as set (refer screenshot) Orion startup logs attached as screenshot