https://prefect.io logo
Powered by
Title
a

Andrew Decker

05/12/2022, 8:37 PM
Hi all, I'm wondering if someone would be able to clarify how flow registration works in the case of 
LocalStorage
and 
KubernetesRun
, as shown in this example. We're trying to follow the same pattern in that example, and our setup registers flows in CI via the Prefect CLI when a PR is merged into the main branch. When doing this, the flow is actually run while building in our CI environment (in this case, I just have a print statement in the flow to verify that it's running). Is there a recommended pattern to register flows in CI without running them?
The flow does run successfully in Prefect, which is progress 🙂
To be more concrete, this is the flow in the 
flows/ingest.py
file 
# This is flows/ingest.py

...

with Flow(
    name=FLOW_NAME,
    storage=Local(
        add_default_labels=False,
        stored_as_script=True,
        path="flows/ingest.py",
    ),
    run_config=KubernetesRun(
        image=f"{ECR_REGISTRY}/{IMAGE}:{IMAGE_TAG}", labels=[IMAGE, ENV, FLOW_NAME]
    ),
    executor=LocalExecutor(),
) as flow:
    print("12345 we are running the flow")
My coworker was trying to use a pattern that wraps the above code in a function 
get()
, and then adds 
if __name__ == "__main__":
    get().run()
to the bottom of the file
However, this yields an error (will post the specific error in a moment - going to rerun to grab it)
a

Anna Geller

05/12/2022, 9:00 PM
is this script in the container image?
generally speaking, this is not a best-practice recipe, I only included it in the repo if you have no other options. What code repository do you use, do you use GitHub or GitLab or sth else?
a

Andrew Decker

05/12/2022, 9:03 PM
Ah, thanks for the feedback. Good to know! We're using GitHub, so I can take a look at the GitHub storage option
Yeah, the script is in the container image
I don't have full context because I didn't set up this pattern on our side, but from what I understand, there was some concern about leaking GitHub secrets. I'll follow up after I've read the GitHub storage doc, but I'm assuming there's a way to do this securely given Prefect's hybrid architecture. Thanks again!
a

Anna Geller

05/12/2022, 9:18 PM
Yup exactly! Check examples here with name: "github_kubernetes_run"
🙏 1
Secrets are stored securely in the Prefect Cloud backend - you can create the needed token here https://github.com/settings/tokens and store it as a Secret in Cloud UI with the name GITHUB_ACCESS_TOKEN and value of the token
a

Andrew Decker

05/12/2022, 9:22 PM
I see, I think there was concern about storing it in Prefect's backend
a

Anna Geller

05/12/2022, 9:22 PM
why?
a

Andrew Decker

05/12/2022, 9:22 PM
...that's a good question
a

Anna Geller

05/12/2022, 9:22 PM
😄
a

Andrew Decker

05/12/2022, 9:22 PM
Let me circle back with an answer
👍 1
😄
a

Anna Geller

05/12/2022, 9:23 PM
it's encrypted and stored in a secure manner
a

Andrew Decker

05/12/2022, 9:31 PM
So the answer I'm getting from DevOps is that ideally we don't want to be sharing our secrets outside of our infrastructure. We incur a bit of risk for every third party we share secrets with, and it's a best practice to limit this as much as possible
Ignoring GitHub vs Local storage for the moment, it looks like both files are set up in the same way, which makes me wonder if the flow will still be run when being registered
a

Anna Geller

05/13/2022, 1:47 AM
I can understand that. In that case, you could leverage local secrets - this page shows how you can set it on your execution layer - ignore that it says how to set Secrets on Server, it works the same for Cloud if you use local secrets and set this variable: 
export PREFECT__CLOUD__USE_LOCAL_SECRETS=true
#prefect-communityJoin Slack
Powered by