https://prefect.io logo
j

Jan Marais

01/25/2021, 2:51 PM
Hi All. I'm new to Prefect and I want to experiment with running a flow on AWS ECS. I'm using this example: https://github.com/lauralorenz/fourteen-oh/blob/main/14-fargate.py, only with S3 storage and the relevant AWS parameter changes. Also making use of a local backend. When running the above script, the flow gets registered successfully and the agent starts. When I run the flow in the UI, it gets submitted and I can also see a ECS task running in AWS console and then it disappears. But the prefect task never starts and the flow never finishes. Any idea what is wrong here? Let me know if I need to share anymore detail. Thanks!
l

Loic M

01/25/2021, 3:04 PM
Have you checked if the registered flows have labels on them ? It might not be your case, but I spent a lot of time to find out that flow registered from my local computer had my computer name on them, and that was why the distant agent - which of course did not have that tag - could not run them
j

Jan Marais

01/25/2021, 3:27 PM
I didn't explicitly give them labels. I understood that the ECS agent would know to pick up flows with the ECSRun run config. Also weird then that the agent log shows the flow run is deployed but that the task never starts
Nevertheless, I will try assigning labels to both agent and flow and see what happens. Thanks for the suggestion
z

Zanie

01/25/2021, 4:04 PM
Hi Jan, if you see the agent deploy the flow run then it shouldn’t be a label issue. Can you provide some logs from the agent? What ECS task do you see get created / can you get logs from it?
j

Jan Marais

01/25/2021, 6:24 PM
Hi Michael. Unfortunately, I don't have the compute instance available to me ATM. This is the only 'log' I can show you from the agent and flow registration:
This is the task definition of the task that I think is running after deployment
z

Zanie

01/25/2021, 8:08 PM
Thanks! If you change the agent logging level
cloud.agent.level
in your config.toml to
DEBUG
we should get a bit more info about where things are going wrong in the deployment.
j

Jan Marais

01/26/2021, 5:25 AM
Thanks, will do so as soon as I have access again.
Sorry @Zanie, is the
cloud.agent.level
applicable with a local backend?
With the DEBUG level:
I have never setup an ECS cluster before so the issue might very well lie there. Do you have a guide that I can follow?
Could it be that the task is executed but never communicated back to the UI?
Interestingly, when I user a Docker agent and run config, the same thing is observed. Flow run gets deployed but the task never gets executed. So maybe not a problem specific to ECS
z

Zanie

01/26/2021, 4:12 PM
Could you share the code for your flow?
j

Jan Marais

01/27/2021, 8:10 AM
It's just:
Copy code
from prefect import Flow, task
from prefect.storage import Docker, S3
import prefect
from prefect.run_configs import ECSRun


@task
def hi():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("HI")


storage = S3("prefect-flows-example")

run_config = ECSRun(
    env={
        "AWS_ACCESS_KEY_ID": "xxx",
        "AWS_SECRET_ACCESS_KEY": "xxx",
    },
    image="prefecthq/prefect:0.14.4-python3.8",
    run_task_kwargs={
        "networkConfiguration": {
            "awsvpcConfiguration": {
                "assignPublicIp": "ENABLED",
                "subnets": ["subnet-087d13d842f616490"],
                "securityGroups": ["sg-073f49dbf01ad5bc9"],
            }
        }
    },
)

with Flow("prefect-test", storage=storage, run_config=run_config) as flow:
    hi()


flow.register("main-project")
I've tried various
ECSRun
configurations, with and without task and execution roles and network configurations.
The agent gets started with:
prefect agent ecs start --cluster prefect-demo
I got the docker agent version working though. Replacing
ECSRun
with
DockerRun
keeping the
env
and
image
parameters the same.
And starting the agent with
prefect agent docker start
My best guess ATM is that I'm not getting this right:
Any guidance on how to deploy the prefect server (either on local machine or ec2 instance) so that ecs can talk to it would be greatly appreciated
Running this exactly as is but using the cloud backend, it works. So I must be missing something when configuring my local server as backend