Kaleb King
06/23/2024, 1:56 AMMarvin
06/23/2024, 1:57 AMpython
from prefect.filesystems import GitHub
block = GitHub(
repository="<https://github.com/my-repo/>",
access_token=<my_access_token> # only required for private repos
)
- Accessing Subfolders: You can specify a subfolder of the repository to access specific parts of it:
python
block.get_directory("folder-in-repo")
### Deployment Use
To use the GitHub storage block in a deployment:
shell
prefect deployment build path/to/flow.py:flow_name --name deployment_name --tag dev -sb github/dev -a
This command-line instruction references the storage block github/dev
which you created earlier.
### Repository Accessibility
- Whole Repository Accessible: When you reference a GitHub repository using a storage block, the whole repository becomes accessible unless you specify particular subdirectories or use .prefectignore
to exclude certain files or directories.
- Exclusion of Files: You can exclude files similarly to .gitignore
by using .prefectignore
.
### References
- Prefect Storage Guide
- Prefect Storage Concepts
If you have further questions or need additional explanations, feel free to ask.Kaleb King
06/23/2024, 3:22 AMprefect build...
command instead of prefect deployment build...
, how can I specify an existing GitHub storage block?Marvin
06/23/2024, 3:22 AMprefect build
command and specify an existing GitHub storage block in Prefect, you'll follow these general steps:
1. Install the Required Libraries:
Make sure to install the relevant library:
sh
pip install prefect-github
2. Register the GitHub Block:
Register the GitHub block with Prefect:
sh
prefect block register -m prefect_github
3. Create and Configure the GitHub Block:
Create a GitHub storage block in the Prefect UI or via code with the repository details:
python
from prefect.filesystems import GitHub
block = GitHub(
repository="<https://github.com/my-repo/>",
access_token=<my_access_token> # only required for private repos
)
block.save("my-github-block-name")
4. Use the Block in the prefect build
Command:
When building the deployment using the prefect build
command, include the GitHub storage block reference:
sh
prefect build path/to/flow.py:flow_name --name deployment_name --tag dev -sb github/my-github-block-name -a
### Example of prefect build
Command
sh
prefect build src/flows/my_flow.py:my_flow_function --name my_deployment --tag dev -sb github/my-github-block-name -a
This command specifies the storage block github/my-github-block-name
which you have created and registered earlier.
### Reference Documentation
- For additional configuration and deeper specifics, you can refer to the Prefect Storage Guide.
Feel free to ask if you have additional questions or require further details.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by