d

    Digo

    2 years ago
    Hi Everyone, How to deploy prefect in AWS ECS?
    j

    josh

    2 years ago
    Hi @Digo while I don’t have an official guide for this you could follow a setup style similar to this tutorial written for GCP. My recommendation for the current iteration of deployment to (which used Docker compose) would be to spin up the server using something like an EC2 instance. More information on deployment strategies coming in the future!
    Also for anyone reading this, here’s the github issue to expand documentation 🙂 https://github.com/PrefectHQ/prefect/issues/2787
    s

    Sandeep Aggarwal

    2 years ago
    Hi @Digo I have had success in deploying complete prefect infrastructure in AWS ECS. Below is what my compose file looks like when running locally. Should be straightforward to adapt same on ECS. Happy to provide more details.
    version: '3.7'
    
    volumes:
        prefect_dbstorage:
    
    services:
        postgres_prefect:
            image: "postgres:11.7"
            ports:
                - "5432:5432"
            environment:
                - POSTGRES_USER=prefect
                - POSTGRES_PASSWORD=prefect
                - POSTGRES_DB=prefect
            restart: "always"
            volumes:
                - prefect_dbstorage:/var/lib/postgresql/data
    
        hasura:
            image: "hasura/graphql-engine:v1.2.1"
            ports:
                - "3000:3000"
            command: "graphql-engine serve"
            environment:
                HASURA_GRAPHQL_DATABASE_URL: <postgresql://prefect:prefect@postgres_prefect:5432/prefect>
                HASURA_GRAPHQL_ENABLE_CONSOLE: "true"
                HASURA_GRAPHQL_SERVER_PORT: "3000"
                HASURA_GRAPHQL_QUERY_PLAN_CACHE_SIZE: 100
            restart: "always"
            depends_on:
                - postgres_prefect
            deploy:
                resources:
                    limits:
                        memory: 200M
    
        graphql:
            image: "prefecthq/server:0.11.5"
            ports:
                - "4201:4201"
            command: bash -c "prefect-server database upgrade -y && python src/prefect_server/services/graphql/server.py"
            environment:
                - PREFECT_SERVER__DATABASE__CONNECTION_URL=<postgresql://prefect:prefect@postgres_prefect:5432/prefect>
                - PREFECT_SERVER__HASURA__HOST=hasura
            restart: "always"
            depends_on:
                - hasura
            deploy:
                resources:
                    limits:
                        memory: 100M
    
        scheduler:
            image: "prefecthq/server:0.11.5"
            command: python src/prefect_server/services/scheduler/scheduler.py
            environment:
                - PREFECT_SERVER__HASURA__HOST=hasura
            restart: "always"
            depends_on:
                - graphql
            deploy:
                resources:
                    limits:
                        memory: 100M
    
        apollo:
            image: "prefecthq/apollo:0.11.5"
            ports:
                - "4200:4200"
            command: "npm run serve"
            environment:
                - HASURA_API_URL=<http://hasura:3000/v1alpha1/graphql>
                - PREFECT_API_URL=<http://graphql:4201/graphql/>
                - PREFECT_API_HEALTH_URL=<http://graphql:4201/health>
            restart: "always"
            depends_on:
                - graphql
            deploy:
                resources:
                    limits:
                        memory: 200M
    
        ui:
            image: "prefecthq/ui:0.11.5"
            ports:
                - "8080:8080"
            command: "/intercept.sh"
            environment:
                - PREFECT_SERVER__GRAPHQL_URL=<http://localhost:4200/graphql>
            restart: "always"
            depends_on:
                - apollo
            deploy:
                resources:
                    limits:
                        memory: 100M
    
        dask_scheduler:
            image: "daskdev/dask"
            command: "dask-scheduler"
            ports:
                - "8787:8787"
                - "8786:8786"
            deploy:
                resources:
                    limits:
                        memory: 100M
    
        dask_worker:
            image: "daskdev/dask"
            command: dask-worker dask_scheduler:8786
            depends_on:
                - dask_scheduler
            volumes:
                - /var/run/docker.sock:/var/run/docker.sock
            deploy:
                replicas: 2
                resources:
                    limits:
                        memory: 1G
    
        agent:
            image: "prefecthq/server:0.11.5"
            command: >4
                bash -c "/wait-for-it.sh apollo:4200
                -- prefect backend server
                && prefect agent start docker
                -v -f --no-docker-interface"
            environment:
                - PREFECT__SERVER__HOST=<http://apollo>
                - PREFECT__SERVER__UI__HOST=<http://ui>
            volumes:
                - /var/run/docker.sock:/var/run/docker.sock
                - ./build/scripts/docker/wait-for-it.sh:/wait-for-it.sh
            depends_on:
                - apollo
                - dask_scheduler
            deploy:
                resources:
                    limits:
                        memory: 100M
    d

    Digo

    2 years ago
    Hi @Sandeep Aggarwal, Thank you sharing the compose file. For me status is changed to Stopped after executing this.
    s

    Sandeep Aggarwal

    2 years ago
    Can you check the logs and see the exact error you got for hasura?
    Nazeer Hussain

    Nazeer Hussain

    2 years ago
    Thanks @Sandeep Aggarwal do you have the task definitions file as well. which Digo might be missing
    ecs-params.yml
    d

    Digo

    2 years ago
    Hi @Sandeep Aggarwal I am getting some version error if i execute the same compose file. My docker version is 19.03.6-ce. Thank you.
    s

    Sandeep Aggarwal

    2 years ago
    Ah. this compose file is just for reference. I don't really use the same compose file for provisioning services on ECS. I have manually created required services to run prefect on ECS. For eg. I have a service to run complete prefect infra i.e. UI, graphql, apollo, agent, scheduler & hasura. Then separate services for dask scheduler and worker resp. The commands to spin up individual containers and environment variables can be referred from this compose file and the actual values will depend on your infrastructure. For eg. I use load balancer to expose UI & apollo servers externally and ECS service discovery for internal communication. Since all prefect infra is running as single service, they can communicate each other using localhost. Hope this helps. Let me know if you have any further queries.
    t

    Thomas Hoeck

    2 years ago
    Hi @Sandeep Aggarwal. What does the wait-for-it.sh contains? 🙂
    s

    Sandeep Aggarwal

    2 years ago
    Hey @Thomas Hoeck its a small script that waits for dependency to reach ready state. The default behaviour of depends_on waits till the service is started. You can find the script here: https://github.com/vishnubob/wait-for-it and read more here: https://docs.docker.com/compose/compose-file/#depends_on. Basically it sort of helps you achieve what you would achieve in ECS using container_dependency
    t

    Thomas Hoeck

    2 years ago
    Ah, that is clever way to get around the shortcomings of depends_on 😄
    Laura Lorenz (she/her)

    Laura Lorenz (she/her)

    2 years ago
    @Marvin archive “Community compose file for prefect server in AWS ECS”
    Marvin

    Marvin

    2 years ago