Hi there folks, how you doing? Github Actions dep...
# ask-community
g
Hi there folks, how you doing? Github Actions deployment question I'm trying to improve our data team workflow with Prefect and our flows are using
Github Storage
. I'm facing a problem when "developing" the flows. Since we're using Github Actions, flows are registered to a
development
project when a Pull Request is opened (and updated). But
Github Storage
will use
main/master
as ref argument. This works fine for production, but won't do while I'm on a development branch. The workaround I've found is to use a pattern like this: • Github Actions Workflow with different steps for Push/Pull Request: ◦ Push to main: generate project folder and register the flow ▪︎ I use
prefect register --project $project --path $path
to do this ◦ Pull Request: register the flow with disabled schedule in
development
project ▪︎ I use the
if __name__ = "__main__"
pattern in order to pass a dynamic
GITHUB_HEAD_REF
to Github Storage ref argument. It's working, but I'm thinking if there is way just use the
prefect cli
for this and avoid writing boilerplate code. The thread has some code snippets and more details!
workflow.yml.yaml
flow.py
w
Can you push the work-in-progress to a branch and just register the flow from that?
g
Note: This is a very opinionated workflow. It uses Poetry and relies on folder structure to automatic project creation.
@Wilson Bilkovich you mean a fixed branch? Like
develop
? The work is already pushed because I'm opening a Pull Request.
But if I run
prefect register --project "$project" --path "$file"
it will use the
main
ref from Github Storage and: • no flow will be found if its the first time (github api 404) • i'll use the main (old) version of the flow In the "production" execution of workflow (after the pull request merge) things will work just fine, because I want to use the
main
ref in my Github Storage. The issue is just within a Pull Request development.
The issue is about automation (and avoiding extra lines of code on each flow). I'm leaving my team tomorrow and they would appreciate this kind of features since our Prefect flow deployment is terrible (EC2 Instance SSH -> Git pull -> python flow.py) But I guess Github Storage might not be the best tool for what I want. Maybe S3 Storage will fare better if I upload a new version of the flow on every commit.