Code!

from prefect import task, Flow
from prefect.run_configs import ECSRun
from prefect.storage import CodeCommit
import requests


@task
def overon_get_missing(days):
    url = "<https://tazl2edzif.execute-api.eu-west-1.amazonaws.com/prod/api/v1/getMissing/>"+ str(days)
    response = <http://requests.post|requests.post>(url)
    missingList = response.json()
    return missingList

task_definition = {
    'networkMode': 'awsvpc',
    'cpu': 1024,
    'memory': 2048,
    'containerDefinitions': [
        {'name': 'ecs-flow'}
    ]
}
run_config = ECSRun(
    run_task_kwargs={'cluster': 'prefect-testing'},
    env={'aws_access_key_id':'xxxxxxxxxxxxxxxx', 
    'aws_secret_access_key':'xxxxxxxxxxxxxx'},
    image='rmansillasoto/prefectecs:v1',
    task_definition=task_definition
)
flow = Flow("ecs-flow",tasks=[overon_get_missing],storage=CodeCommit(repo="prefect_flows", path="flows/test_ecs.py", commit = "master"),
            run_config=run_config)

if __name__ == "__main__":
    
    flow.run(days=30)

I also ran an ECSAgent which didn´t show any error during the flow’s execution, just Found 1 flow….and Deploying flow run….:

from prefect.agent.ecs.agent import ECSAgent

AGENT = ECSAgent(cluster="prefect-testing", region_name='eu-west-1', run_task_kwargs_path='./config.yaml', launch_type='FARGATE', labels=['ecs', 'fargate'],aws_access_key_id='xxxxxxxxxxxxxxxxx', aws_secret_access_key='xxxxxxxxxxxxxxx')

AGENT.start()

____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2021-06-01 15:46:05,196] INFO - agent | Starting ECSAgent with labels ['ecs', 'fargate']
[2021-06-01 15:46:05,196] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2021-06-01 15:46:05,196] INFO - agent | Agent connecting to the Prefect API at <http://10.10.5.107:4200>
[2021-06-01 15:46:05,201] INFO - agent | Waiting for flow runs...
[2021-06-01 15:46:05,236] INFO - agent | Found 1 flow run(s) to submit for execution.
[2021-06-01 15:46:05,282] INFO - agent | Deploying flow run '58aa8ee4-69af-41a9-9236-dc75b24d8e6e'
[2021-06-01 15:46:41,203] INFO - agent | Found 1 flow run(s) to submit for execution.
[2021-06-01 15:46:41,243] INFO - agent | Deploying flow run '8c1096dc-f5c4-4633-aeb6-0afcc0105a4d'
[2021-06-01 15:48:47,446] INFO - agent | Found 1 flow run(s) to submit for execution.
[2021-06-01 15:48:47,488] INFO - agent | Deploying flow run '1a1a2a05-e850-403d-aa9c-15b00b785b7e'
[2021-06-01 15:50:43,585] INFO - agent | Found 1 flow run(s) to submit for execution.
[2021-06-01 15:50:43,630] INFO - agent | Deploying flow run '1edfdd0d-5777-42fd-b866-5978603467f1'
[2021-06-01 15:58:50,667] INFO - agent | Found 1 flow run(s) to submit for execution.
[2021-06-01 15:58:50,706] INFO - agent | Deploying flow run '58aa8ee4-69af-41a9-9236-dc75b24d8e6e'

I´m using right now user´s credentials with ECSFull and ECSTask policies….also CodecommitReadOnly and the prefect’s container image has all del packages needed for the flows….

does anyone have any clue? Thanks!

yes

sorry, I´ll move them

let me try

It seems like the storage is not being respected here. Could you try

S3Storage

first like this and see if it works? https://docs.prefect.io/orchestration/execution/storage_options.html#aws-s3

Oh that’s weird. How are you registering this one?

i forgot to mention it…

prefect register --project raulTesting -p ecs_test.py -l "ecs" -l "fargate" --force

Is this a DockerHub image?

Seems like it should work. You can try formatting your task-defnition like this

Ok so I got it working. I hit the same error as you. That error message is deceiving. I think it is related to not being able to find the cluster. So I’ll give you the code and exact steps I had to do after that error to get things working

Code:

from prefect import task, Flow
from prefect.run_configs import ECSRun
from prefect.storage.codecommit import CodeCommit

@task
def overon_get_missing(days):
    return days

task_definition = {
    'networkMode': 'awsvpc',
    'cpu': 1024,
    'memory': 2048,
    'containerDefinitions': [
        {'name': 'flow'}
    ]
}

run_config = ECSRun(
    run_task_kwargs={'cluster': 'my-fargate-cluster'},
    env={'AWS_ACCESS_KEY_ID':'XXXXXXXXXXXXX', 
    'AWS_SECRET_ACCESS_KEY':'XXXXXXXXXXXXX',
    'AWS_DEFAULT_REGION': 'us-east-2'},
    image='prefecthq/prefect:latest-python3.8',
    task_definition=task_definition
)

with Flow("ecs-flow") as flow:
    overon_get_missing(30)

flow.storage = CodeCommit(repo="prefect_flows", path="flows/test_ecs.py", commit="main")

flow.run_config = run_config

flow.register('omlds')

1. Upload this to CodeCommit at the specified path 2. Register with

python filename.py

3. Create an ECS cluster with

aws ecs create-cluster --cluster-name my-fargate-cluster --capacity-providers FARGATE

4. Note that the cluster-name in step 3 matches the

ECSRun

cluster name in

run_task_kwargs

5. Start an agent with something like

prefect agent ecs start --cluster arn:aws:ecs:us-east-2:XXXXXXXXXXXXX:cluster/my-fargate-cluster

6. My credentials passed to ECSRun have admin access for this 7. Quick Run from the UI

I am not 100% sure, but I believe the

Container.image should not be null

might be related to not having a cluster or not being able to find the cluster. If you have a cluster, it could also be not having the right capacity-providers