Hi everone. Are there any best practices for running flows locally that will ultimately run on Kubernetes? I was running them using the docker agent but I am starting to see a bunch of differences between the local environment and the remote kubernetes cluster. One of of these differences is how secrets are handled. The approach the devops person asked me to use is mounting the secrets as a volume in a custom job template that I pass to the

KubernetesRun

run config. I wrote a task that reads the secrets from the file system. However, locally, I was using environment variables for secrets. I'll probably have to see a way to mount the secrets locally on the docker container but before I go too far down this path, I'd like to get some input from the community. I suppose I could try to set up Kubernetes locally but I know very little about Kubernetes. This is my first experience with it.

Hey @Pedro Machado, have you seen the KubernetesTask secret ? But yeah, I guess to test, you’d want to setup Kubernetes locally as Docker won’t be reflective.

I did but that would require an API key which would have to be stored in Prefect Cloud, correct? We were trying to keep all secrets in the execution environment.

Maybe not. Looking at the docs.

Attempt to use a Prefect Secret that contains a Kubernetes API Key. If kubernetes_api_key_secret = None then it will attempt the next two connection methods. By default the value is KUBERNETES_API_KEY so providing None acts as an override for the remote connection. 2. Attempt in-cluster connection (will only work when running on a Pod in a cluster) 3. Attempt out-of-cluster connection using the default location for a kube config file

@Pedro Machado we are using kubernetessecret task and localenvsecret. When we setup the flow we are switching between one to the other. This allow us to do local dev and production as well using the same flow. I'm on mobile so it is not simple to put a coffee sample Let me know if you need

Thanks @davzucky! Did you mean EnvVarSecret? How are you switching between the two?

we are doing that at flow build time. This is a sample that shows how we do that

import os
import distutils.util
from prefect.tasks.kubernetes import KubernetesSecret
from prefect.tasks.secrets import EnvVarSecret
from prefect import Flow, task


@task
def print_connection_string(connection_string: str):
    print(f"Connection string => {connection_string}")


def get_flow(use_local_secret: bool = False) -> Flow:
    with Flow("Sample Secret") as flow:
        kSecret = KubernetesSecret(kubernetes_api_key_secret="")
        con_str = (
            EnvVarSecret("MONGO_URL")
            if use_local_secret
            else kSecret(secret_name="k8s_secrets", secret_key="MONGO_URL")
        )
        print_connection_string(con_str)

    return flow


use_local_secret = bool(
    distutils.util.strtobool(os.environ.get("USE_LOCAL_SECRET", "False"))
)
flow = get_flow(use_local_secret)

flow.run()

Great. I'll try something similar. Thanks!