Hi All. I'm new to Prefect and I want to experiment with running a flow on AWS ECS. I'm using this example: https://github.com/lauralorenz/fourteen-oh/blob/main/14-fargate.py, only with S3 storage and the relevant AWS parameter changes. Also making use of a local backend. When running the above script, the flow gets registered successfully and the agent starts. When I run the flow in the UI, it gets submitted and I can also see a ECS task running in AWS console and then it disappears. But the prefect task never starts and the flow never finishes. Any idea what is wrong here? Let me know if I need to share anymore detail. Thanks!

Have you checked if the registered flows have labels on them ? It might not be your case, but I spent a lot of time to find out that flow registered from my local computer had my computer name on them, and that was why the distant agent - which of course did not have that tag - could not run them

I didn't explicitly give them labels. I understood that the ECS agent would know to pick up flows with the ECSRun run config. Also weird then that the agent log shows the flow run is deployed but that the task never starts

Hi Jan, if you see the agent deploy the flow run then it shouldn’t be a label issue. Can you provide some logs from the agent? What ECS task do you see get created / can you get logs from it?

Hi Michael. Unfortunately, I don't have the compute instance available to me ATM. This is the only 'log' I can show you from the agent and flow registration:

Thanks! If you change the agent logging level

cloud.agent.level

in your config.toml to

DEBUG

we should get a bit more info about where things are going wrong in the deployment.

Thanks, will do so as soon as I have access again.

Could you share the code for your flow?

It's just:

from prefect import Flow, task
from prefect.storage import Docker, S3
import prefect
from prefect.run_configs import ECSRun


@task
def hi():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("HI")


storage = S3("prefect-flows-example")

run_config = ECSRun(
    env={
        "AWS_ACCESS_KEY_ID": "xxx",
        "AWS_SECRET_ACCESS_KEY": "xxx",
    },
    image="prefecthq/prefect:0.14.4-python3.8",
    run_task_kwargs={
        "networkConfiguration": {
            "awsvpcConfiguration": {
                "assignPublicIp": "ENABLED",
                "subnets": ["subnet-087d13d842f616490"],
                "securityGroups": ["sg-073f49dbf01ad5bc9"],
            }
        }
    },
)

with Flow("prefect-test", storage=storage, run_config=run_config) as flow:
    hi()


flow.register("main-project")

I've tried various

ECSRun

configurations, with and without task and execution roles and network configurations.