<@ULVA73B9P> I want to create a prefect integratio...
# marvin-ai
a
@Marvin I want to create a prefect integration which has a reusable block, flow and deployment that is open source and on github so that others can use it, and easily install my flow/deployment into their apps. Is there a template or example for the project i could use to base my work on? Ideally with github actions as a CI to automatically package and publish the python module? And is there a central library/list of prefect integrations where it could be listed so it is discoverable by other people?
m
thought for 34.3 seconds
Great news! I can help you with creating a Prefect integration. Here's a comprehensive guide based on the available information: 1. Template Repository There is an official template you can use: prefect-collection-template. This template includes: - Basic project structure - GitHub Actions for CI/CD - Package publishing setup - Example blocks, flows, and tasks - Testing infrastructure 2. Getting Started To use the template: 1. Visit https://github.com/PrefectHQ/prefect-collection-template 2. Click "Use this template" to create your repository 3. Follow the quickstart guide in the README 3. Key Components to Include - Custom blocks (if needed) - Reusable flows and tasks - Documentation - Tests - GitHub Actions workflows (included in template) - PyPI publishing setup 4. Discoverability Prefect integrations can be discovered through: - Prefect Integrations Catalog - To get your integration listed: - Ensure your package name starts with
prefect-
- Follow the contribution guidelines - Your package will be automatically discovered once published to PyPI 5. Best Practices - Use semantic versioning - Include clear documentation - Provide usage examples - Add type hints - Include tests - Use the standard
prefect-
naming convention for your package 6. Installation for Users Once published, users can install your integration using pip:
Copy code
bash
pip install prefect-yourintegration
The flows and blocks will then be automatically registered and available in their Prefect environment. Would you like me to dive deeper into any particular aspect of creating a Prefect integration? For example: - Setting up GitHub Actions for automated testing and publishing - Structuring your blocks and flows - Writing documentation - Testing strategies Let me know what specific area you'd like to focus on first!
a
@Marvin What is a "prefect collection"?
m
thought for 34.0 seconds
Let me clarify this terminology situation, as it's undergone some evolution: 1. Historical Context: - Originally, "collections" was the term used for packages that extended Prefect's functionality - These were essentially plugins or add-ons that provided additional tasks, flows, and utilities 2. Current Terminology: - The preferred term now is "integrations" rather than "collections" - This is reflected in the official documentation - Even though the template repository is still called "prefect-collection-template", it's used to create what are now called "integrations" 3. What They Actually Are: An integration (formerly called a collection) is: - A Python package that extends Prefect's functionality - Usually focuses on connecting Prefect with a specific service or tool - Contains reusable components like: - Custom blocks for storing credentials or configuration - Pre-built tasks for common operations - Example flows showing how to use the integration - Utilities specific to the integration's purpose 4. Examples: Some popular Prefect integrations include: -
prefect-aws
for Amazon Web Services -
prefect-gcp
for Google Cloud Platform -
prefect-slack
for Slack notifications -
prefect-dbt
for dbt integration The term "collection" is essentially legacy terminology, and "integration" is the more current and preferred term. When creating your own extension to Prefect, you should think of it and refer to it as an "integration", even though you might see references to "collections" in some template names or older documentation. Would you like me to provide more specific details about any aspect of creating a Prefect integration or how they work in practice?