haven

    haven

    1 year ago
    KubernetesSecret question: currently it seems that it's designed such that
    KubernetesSecret
    should not rely on any upstream task/parameters. However, I think it might not be a great assumption. I'm using an
    env
    parameter that denotes
    "DEV"
    or
    "PROD"
    and then would decide what secret I want to retrieve from k8s later (usually a database secret). Any plan to allow
    KubernetesSecret
    to be a dynamic task and interpreted naturally in the flow without having to manually use
    flow.set_dependencies
    ?
    Ah it seems setting the dependencies isn't enough. Would appreciate a suggestion!
    davzucky

    davzucky

    1 year ago
    We are using it in the same use case that you have? Can you share your code?
    haven

    haven

    1 year ago
    It's something like this
    env = Parameter("env", default="DEV")
        dt = Parameter("dt", default=date.today().strftime("%Y-%m-%d"))
        use_proxy = Parameter("use_proxy", default=False)
    
        env_config = config(env)
        db = env_config["db"]
        db_password = KubernetesSecret(db, "password", kubernetes_api_key_secret=None)
        db_user = KubernetesSecret(db, "username", kubernetes_api_key_secret=None)
        db_host = KubernetesSecret(db, "host", kubernetes_api_key_secret=None)
    where
    @task
    def config(env: str) -> Dict[str, Any]:
        ...
    davzucky

    davzucky

    1 year ago
    You need to create an instance of kubernetessecret first and use that instance to pass the parameters
    Will do a sample when I'm on a computer of you cannot make it work
    The way you are using it is at flow built time that it is setup not at run time
    Hope that help
    env = Parameter("env", default="DEV")
        dt = Parameter("dt", default=date.today().strftime("%Y-%m-%d"))
        use_proxy = Parameter("use_proxy", default=False)
        env_config = config(env)
        db = env_config["db"]
        K8sSecret = KubernetesSecret(kubernetes_api_key_secret=None)
        db_password = K8sSecret(secret_name=db, secret_key="password")
        db_user = K8sSecret(secret_name=db, secret_key="username")
        db_host = K8sSecret(secret_name=db, secret_key="host")
    Try that
    haven

    haven

    1 year ago
    that works like a charm. thanks a lot! @davzucky
    davzucky

    davzucky

    1 year ago
    Welcome. Good continuation on your exploration