I cannot run a simple flow on ECS. I have setup a simple flow like this

import prefect
from prefect import task, Flow
from prefect.run_configs import ECSRun


@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello World!")
    return 0


task_definition = {
    'networkMode': 'awsvpc',
    'cpu': 1024,
    'memory': 2048,
    'containerDefinitions': [
        {'name': 'flow'}
    ]
}


run_config = ECSRun(
    image='prefecthq/prefect',
    task_definition=task_definition
)

flow = Flow("hello-flow",
            tasks=[hello_task],
            run_config=run_config)

flow.register(project_name='Prefect_Tutorials', labels=['aws', 'ecs', 'fargate'])

The ECS agent can successfully trigger an ECS task. However, the flow run failed with error

Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/Users/kha/'")

. Why does it include something specific to my name in the flow?

Hi @Kha Nguyen I suspect this is because you did not define a

Stoage

in the flow. The storage is the location the agent will pull your Flow from. A lot of people use S3 storage with ECS. The default storage is LocalStorage and saves it in a default path. When your agent picks up the flow, it is looking for the flow in the default path, which is your home directory.

Ah I see. Thanks Kevin, I will setup the storage

I think it’s more about the credentials you use having access to S3.

The agent is running locally, and it has superadmin access to AWS

you need them as environment variables on the agent

I already specified the credential, but it still cannot locate credentials

Looking into this

It seems it is not from agent. It came from running this command on an ECS task.

I’m not 100% sure where this is coming from at the moment. Could you try adding the environment variables first through the

ECSRun

on the flow side with

ECSRun(…, env={'AWS_ACCESS_KEY_ID': 'aaa', 'AWS_SECRET_ACCESS_KEY': 'bbb'

}

I tried your suggestion. It is still failing in the same way

Let me test it on my end and get back to you

Yes, the file is visible in S3

Thanks!

It is the role policy issue. I set the

task_role_arn

to a role that has access to S3 bucket, and it works. Now I have another error from ECS task

Failed to load and execute Flow's environment: StorageError("An error occurred while unpickling the flow:\n  TypeError('code() takes at most 15 arguments (16 given)')\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n  - python: (flow built with '3.9.2', currently running with '3.7.10')")

This is image matching yes. We don’t have a 3.9 image and the default image gets 3.7. Images here .

Thanks Kevin

Glad you got it working 🙂