Is the error you’re seeing something like this?

And I am on prefect 0.14.12 btw

Where is your Agent running?

Additionally, just curious if this is the optimal way to find a solution to my problem. Trying to run some large parallelized Flows. Since Dask handles the execution across many nodes, I figured I could use Fargate to provide a bunch of containers and thus, that nice Prefect map function can run across many containers and get some good speed ups.

Wondering if you’ve seen better ways to scale Flows in this manner.

Yes this sounds like the way to go to scale your flows

Is your flow using a lot of memory from the start?

Does you agent pick up the flow (local command line?)

https://aws.amazon.com/premiumsupport/knowledge-center/ecs-tasks-stuck-pending-state/ this says you may need to increase container instance as well

okay ill give that a shot.

I will mention though that debugging ECS is a pain point on our front. There was someone with different issues last Monday. This is on our radar and we intend to produce a minimum working example in a blog post in the next week.

I tried with Prefect Cloud instead of Prefect Server.

This was the task definition that Prefect generated:

{
  "ipcMode": null,
  "executionRoleArn": "arn:aws:iam::XXXXX:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": null,
      "entryPoint": null,
      "portMappings": [],
      "command": null,
      "linuxParameters": null,
      "cpu": 0,
      "environment": [
        {
          "name": "PREFECT__CONTEXT__IMAGE",
          "value": "prefecthq/prefect:latest-python3.6"
        }
      ],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "prefecthq/prefect:latest-python3.6",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "flow"
    }
  ],
  "placementConstraints": [],
  "memory": "2048",
  "taskRoleArn": null,
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "taskDefinitionArn": "arn:aws:ecs:us-west-2:XXXXXXX:task-definition/prefect-hello-flow:4",
  "family": "prefect-hello-flow",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "1024",
  "revision": 4,
  "status": "INACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": [],
  "statusString": "(INACTIVE)"
}

I “xxx’d” out our aws account number.

It’s the default ecs task execution role in IAM:

prefect agent ecs start --cluster arn:aws:ecs:us-west-2:XXXXX:cluster/prefect-cluster --execution-role-arn arn:aws:iam::XXXXX:role/ecsTaskExecutionRole --label Raptor-Macbook-Raptor-2.local

And here’s how I started my ECS agent. ^ I think with those messages you have everything you need to help us debug.

I think it’s just throwing a “ModuleNotFoundError” the moment the container spins up, but I have no idea where that is getting set or needs to be overrode.

^ and it’s the image from prefect. No modifications.

Here’s a better screenshot of Prefect Cloud. Realized that was a small screenshot I previously sent.

I assume the agent is running on the same machine that registered the flow?

Hi @Eddie Obropta, the error we’re seeing here happens because there is a default storage called

LocalStorage

. By default LocalStorage will save your serialized flow in Users/username/. If you combine

LocalRun

with

LocalStorage

, what happens is you’re telling Prefect that the serialized script lives in that Users/username location. The agent will find that serialized Flow and unpickle it and run it.

So now what happens is that we’re using the default

LocalStorage

along with the

ECSRun

config. It downloads the image and tries to find that serialized flow inside the Docker container. But it doesn’t live there because it’s on

LocalStorage

. For your specific example, we need to configure a

Storage

that your Agent can each. Since you’re already on AWS, we recommend S3 Storage . Specifying the S3 storage will store your script when you register the flow. Now your ECSRun will be able to grab that script correctly (the error is because it incorrectly tried to grab it from Users/username/)

I have spoken with the core team and I will be creating an issue to have more descriptive messages around the usage of the default

LocalStorage

.

And I haven’t forgotten about Matthew! Will ping him when I get a tutorial together 😅