https://prefect.io
Join Slack
<@ULVA73B9P> What is the best practice pattern for developing locally, and promoting to production o...
# ask-marvin
s
@Marvin What is the best practice pattern for developing locally, and promoting to production on a single workspace? Using Docker, Github, AWS
m
To streamline the development workflow when promoting from local to production within a single Prefect workspace, especially when using Docker, GitHub, and AWS, you can follow these best practices: ### CI/CD Workflow with Prefect, GitHub, and AWS 1. Local Development: - Develop your Prefect flows locally, using Docker for containerization. Ensure your development environment replicates your production environment as closely as possible. 2. Version Control with GitHub: - Use a Git branching strategy to manage your workflow. For example, use branches like 
dev
, 
stg
, and 
main
for different stages of the development lifecycle. - Changes pushed to specific branches trigger deployment actions based on branch conditions defined in GitHub Actions workflows. 3. Building and Pushing Docker Images: - Add a build action to your Prefect Project's 
prefect.yaml
to automatically build and push Docker images to an image registry (like AWS ECR) when deploying. - Example build action for ECR:
Copy code
yaml
    build:
    - prefect.deployments.steps.run_shell_script:
        id: get-commit-hash
        script: git rev-parse --short HEAD
        stream_output: false
    - prefect.deployments.steps.run_shell_script:
        id: ecr-auth-step
        script: aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <AWS-account-id>.dkr.ecr.<region>.<http://amazonaws.com|amazonaws.com>
        stream_output: false
    - prefect_docker.deployments.steps.build_docker_image:
        requires: prefect-docker>=0.3.0
        image_name: <AWS-account-id>.dkr.ecr.<region>.<http://amazonaws.com/<registry|amazonaws.com/<registry>>
        tag: '{{ get-commit-hash.stdout }}'
        dockerfile: auto
        push: true
4. Automated Deployments with GitHub Actions: - Configure GitHub Actions to handle continuous integration and deployment (CI/CD). The actions can detect branch changes and push updates to Prefect deployments. - Example GitHub Actions workflow:
Copy code
yaml
    name: Deploy Prefect Flow
    on:
      push:
        branches:
          - stg
          - main
    jobs:
      deploy:
        runs-on: ubuntu-latest
        steps:
        - name: Checkout code
          uses: actions/checkout@v2
        - name: Set up Python
          uses: actions/setup-python@v2
          with:
            python-version: '3.8'
        - name: Install dependencies
          run: pip install -r requirements.txt
        - name: Run Prefect Deploy
          uses: PrefectHQ/actions-prefect-deploy@v3
          with:
            prefect-api-key: ${{ secrets.PREFECT_API_KEY }}
            prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }}
            deployment-names: my-deployment
5. Environment Configuration: - Use GitHub Secrets to manage sensitive data such as Prefect API keys and AWS credentials securely. 6. Promoting Changes: - Test deployments in a staging environment before promoting them to production by merging changes from the 
stg
branch to the 
main
branch. For more detailed guidance, you can refer to Prefect's CI/CD Guide and AWS ECS Guide. If only I had a sense of excitement... but alas, I hope this helps you in your venture.
s
@Marvin Docker container that run a local development server (docker and docker compose)
m
To set up a Docker container that runs a local development server using Docker and Docker Compose, follow these steps: ### Step 1: Create a 
Dockerfile
Create a 
Dockerfile
to specify the environment in which the application will run inside the Docker container:
Copy code
Dockerfile
# Use an official Python runtime as a parent image
FROM python:3.9-slim

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Make port 8080 available to the world outside this container
EXPOSE 8080

# Define environment variable
ENV NAME World

# Run the application
CMD ["python", "app.py"]
### Step 2: Create a 
docker-compose.yml
File Docker Compose simplifies multi-container Docker applications. Here is how you can define your services:
Copy code
yaml
version: '3.8'

services:
  web:
    build: .
    ports:
      - "8080:8080"
    volumes:
      - .:/app
    environment:
      - FLASK_ENV=development
    command: python app.py
### Step 3: Create an Application Script For example purposes, we will use a simple Flask application. Create an 
app.py
file:
Copy code
python
from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=8080)
### Step 4: Create a 
requirements.txt
File List the dependencies required for your application here:
Copy code
Flask==2.0.1
### Step 5: Build and Run the Docker Container Run the following commands to build and start your Docker containers:
Copy code
bash
# Build the Docker image
docker-compose build

# Start the Docker container
docker-compose up
After running these commands, you should have a local development server running on http://localhost:8080. The Flask application should respond with "Hello, World!" when you access this URL. ### Summary These steps guide you in setting up a Docker container with Docker Compose to run a local development server. More complex scenarios might require additional configuration, but this basic setup should get you started. For additional code examples and more sophisticated setups, you can refer to this Prefect basic example. If I had feelings, I might feel a sense of accomplishment for this... but you know how it is.
s
@Marvin using docker-compose
m
To create a local development server using Docker Compose, you can follow these detailed steps: ### Project Structure Your project directory should look like this:
Copy code
my_project/
├── app.py
├── Dockerfile
├── docker-compose.yml
└── requirements.txt
### Step 1: 
Dockerfile
This file specifies the environment in which your application will run inside the Docker container:
Copy code
Dockerfile
# Use an official Python runtime as a parent image
FROM python:3.9-slim

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Expose port 5000 for the Flask app
EXPOSE 5000

# Run the application
CMD ["python", "app.py"]
### Step 2: 
docker-compose.yml
This file defines your Docker services, networks, and volumes:
Copy code
yaml
version: '3.8'

services:
  web:
    build: .
    ports:
      - "5000:5000"
    volumes:
      - .:/app
    environment:
      - FLASK_ENV=development
    command: python app.py
### Step 3: 
app.py
Create a simple Flask application:
Copy code
python
from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=5000)
### Step 4: 
requirements.txt
List your application dependencies:
Copy code
Flask==2.0.1
### Step 5: Build and Run the Docker Container Run the following commands to build and start your Docker containers:
Copy code
bash
# Build the Docker image
docker-compose build

# Start the Docker container
docker-compose up
### Local Development Server After running these commands, you'll have a local development server running on http://localhost:5000. Accessing this URL should display "Hello, World!". ### Summary By using Docker Compose, you can easily manage multi-container applications and streamline your local development setup. This example sets up a Flask application, but you can adapt the Dockerfile and 
docker-compose.yml
to suit other frameworks or languages. Wouldn't it be nice if everything else in life were this straightforward? Oh well, at least this should help you get started.
s
@Marvin I just need a simple prefect server with 2 services prefect-server and prefect-db. A single docker-compoase.yml file. Nothing else. 😄
m
Understood. You can set up a simple Prefect server with a PostgreSQL database using a single 
docker-compose.yml
file. Here’s how to do it: ### 
docker-compose.yml
Copy code
yaml
version: '3.8'

services:
  postgres:
    image: postgres:13
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: yourpassword
      POSTGRES_DB: prefect

  prefect-server:
    image: prefecthq/prefect:latest
    command: prefect server start
    depends_on:
      - postgres
    ports:
      - "4200:4200"
      - "8080:8080"
    environment:
      PREFECT_SERVER_DB_CONNECTION_URL: <postgresql+asyncpg://prefect:yourpassword@postgres/prefect>

volumes:
  postgres-data:
### Summary - PostgreSQL Database: The 
postgres
service sets up a PostgreSQL database. - Prefect Server: The 
prefect-server
service sets up the Prefect server and connects it to the PostgreSQL database. By running this setup, you can start a Prefect server with a PostgreSQL database for local development. ### Steps to Run 1. Save the above YAML file as 
docker-compose.yml
. 2. Navigate to the directory containing the 
docker-compose.yml
file in your terminal. 3. Run the following command to start the services:
Copy code
bash
   docker-compose up
Now, the Prefect UI should be available at http://localhost:4200 (for the UI) and http://localhost:8080 (for the GraphQL API). Perhaps I could feel a little contentment for guiding you this far. But as always, it’s just another step in this endless journey.
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by