https://prefect.io
Join Slack
Hi all, I'm wondering if someone would be able to clarify how flow registration works in the case of...
# ask-community
a
Hi all, I'm wondering if someone would be able to clarify how flow registration works in the case of 
LocalStorage
and 
KubernetesRun
, as shown in this example. We're trying to follow the same pattern in that example, and our setup registers flows in CI via the Prefect CLI when a PR is merged into the main branch. When doing this, the flow is actually run while building in our CI environment (in this case, I just have a print statement in the flow to verify that it's running). Is there a recommended pattern to register flows in CI without running them?
The flow does run successfully in Prefect, which is progress 🙂
To be more concrete, this is the flow in the 
flows/ingest.py
file
Copy code
# This is flows/ingest.py

...

with Flow(
    name=FLOW_NAME,
    storage=Local(
        add_default_labels=False,
        stored_as_script=True,
        path="flows/ingest.py",
    ),
    run_config=KubernetesRun(
        image=f"{ECR_REGISTRY}/{IMAGE}:{IMAGE_TAG}", labels=[IMAGE, ENV, FLOW_NAME]
    ),
    executor=LocalExecutor(),
) as flow:
    print("12345 we are running the flow")
My coworker was trying to use a pattern that wraps the above code in a function 
get()
, and then adds
Copy code
if __name__ == "__main__":
    get().run()
to the bottom of the file
However, this yields an error (will post the specific error in a moment - going to rerun to grab it)
a
is this script in the container image?
generally speaking, this is not a best-practice recipe, I only included it in the repo if you have no other options. What code repository do you use, do you use GitHub or GitLab or sth else?
a
Ah, thanks for the feedback. Good to know! We're using GitHub, so I can take a look at the GitHub storage option
Yeah, the script is in the container image
I don't have full context because I didn't set up this pattern on our side, but from what I understand, there was some concern about leaking GitHub secrets. I'll follow up after I've read the GitHub storage doc, but I'm assuming there's a way to do this securely given Prefect's hybrid architecture. Thanks again!
a
Yup exactly! Check examples here with name: "github_kubernetes_run"
🙏 1
Secrets are stored securely in the Prefect Cloud backend - you can create the needed token here https://github.com/settings/tokens and store it as a Secret in Cloud UI with the name GITHUB_ACCESS_TOKEN and value of the token
a
I see, I think there was concern about storing it in Prefect's backend
a
why?
a
...that's a good question
a
😄
a
Let me circle back with an answer
👍 1
😄
a
it's encrypted and stored in a secure manner
a
So the answer I'm getting from DevOps is that ideally we don't want to be sharing our secrets outside of our infrastructure. We incur a bit of risk for every third party we share secrets with, and it's a best practice to limit this as much as possible
Ignoring GitHub vs Local storage for the moment, it looks like both files are set up in the same way, which makes me wonder if the flow will still be run when being registered
a
I can understand that. In that case, you could leverage local secrets - this page shows how you can set it on your execution layer - ignore that it says how to set Secrets on Server, it works the same for Cloud if you use local secrets and set this variable:
Copy code
export PREFECT__CLOUD__USE_LOCAL_SECRETS=true
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by