Moritz Rocholl
08/21/2024, 6:27 PMMarvin
08/21/2024, 6:28 PM# Use the official Python base image
FROM python:3.9-slim
# Set the working directory
WORKDIR /app
# Copy the requirements file
COPY requirements.txt .
# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt
# Copy the rest of the application code
COPY . .
# Set the entrypoint for the container
ENTRYPOINT ["python", "my_flow_script.py"]
Make sure your flow script is in the same directory and is named my_flow_script.py
. Also, create a requirements.txt
file containing:
requirements.txt:
prefect
httpx
# Add any other dependencies required by your flow
Here's a basic example of a Prefect flow script you can use:
my_flow_script.py:
python
from prefect import flow
import httpx
@flow(log_prints=True)
def get_repo_info(repo_name: str = "PrefectHQ/prefect"):
url = f"<https://api.github.com/repos/{repo_name}>"
response = httpx.get(url)
response.raise_for_status()
repo = response.json()
print(f"{repo_name} repository statistics 🎉:")
print(f"Stars 🌟 : {repo['stargazers_count']}")
print(f"Forks 🍴 : {repo['forks_count']}")
print(f"Open issues 🐞 : {repo['open_issues_count']}")
if __name__ == "__main__":
get_repo_info()
To build the Docker image, run in your terminal:
docker build -t my_prefect_flow .
For more detailed information, refer to the Prefect documentation on Docker.Moritz Rocholl
08/21/2024, 6:31 PMMoritz Rocholl
08/21/2024, 6:32 PMMarvin
08/21/2024, 6:32 PMget_repo_info.deploy()
, you'll need to ensure that the Docker image you build can execute Prefect flows and is set up correctly for the deployment.
Here's an updated approach:
1. Custom Dockerfile:
This Dockerfile includes all necessary steps to create an environment compatible with Prefect.
Dockerfile:
# Use the official Python base image
FROM python:3.9-slim
# Set the working directory
WORKDIR /app
# Copy the requirements file
COPY requirements.txt .
# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt
# Copy the rest of the application code
COPY . .
# Set the entrypoint for the container
ENTRYPOINT ["python", "-m", "prefect.engine"]
2. requirements.txt:
Ensure the required dependencies are listed here.
requirements.txt:
prefect
httpx
# Add any other dependencies required by your flow
3. Flow Script (my_flow_script.py):
Update your flow script to include the deployment part.
my_flow_script.py:
python
from prefect import flow
import httpx
@flow(log_prints=True)
def get_repo_info(repo_name: str = "PrefectHQ/prefect"):
url = f"<https://api.github.com/repos/{repo_name}>"
response = httpx.get(url)
response.raise_for_status()
repo = response.json()
print(f"{repo_name} repository statistics 🎉:")
print(f"Stars 🌟 : {repo['stargazers_count']}")
print(f"Forks 🍴 : {repo['forks_count']}")
print(f"Open issues 🐞 : {repo['open_issues_count']}")
if __name__ == "__main__":
get_repo_info.deploy(
name="my-first-deployment",
work_pool_name="my-docker-pool",
image="my-first-deployment-image:tutorial",
push=False # Set to True if you need to push to a remote registry
)
4. Build the Docker Image:
Use the docker build
command to create your Docker image.
docker build -t my-first-deployment-image:tutorial .
This setup ensures that your custom Dockerfile is compatible with get_repo_info.deploy()
. By deploying your flow using Flow.deploy()
, Prefect can handle running the flow inside a Docker container with the specified image. If Prefect is running a Docker worker on your local machine, you can set push=False
. Otherwise, ensure to push your image to a registry accessible by Prefect workers.
For more detailed and updated instructions, refer to the Prefect documentation on Docker and Prefect flow deployment.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by