m

    Michail Melonas

    1 year ago
    I am having trouble passing sensitive information to my flow's run configuration. Having:
    secret = PrefectSecret("SOME_SECRET")
    flow.storage = S3(bucket="some-bucket", secrets=["AWS_CREDENTIALS"])
    
    flow.run_config = KubernetesRun(
        image="<http://registry.gitlab.com/<some_image>|registry.gitlab.com/<some_image>>",
        labels=["prefect-14"],
        env={
            "PREFECT__LOGGING__LEVEL": "DEBUG",
            "SECRET": secret
        },
        image_pull_secrets=["image-puller"],
    )
    and then running:
    flow.register("ProjectName")
    I get:
    TypeError: Object of type PrefectSecret is not JSON serializable
    Any thoughts on this?
    Kevin Kho

    Kevin Kho

    1 year ago
    Secrets need to be JSON Serializable. Is your secret a list or dict?
    m

    Michail Melonas

    1 year ago
    It's a string saved on Prefect Cloud
    Kevin Kho

    Kevin Kho

    1 year ago
    I think if it’s outside of the Flow, you need to do
    PrefectSecret(…).run()
    . Could you try that?
    m

    Michail Melonas

    1 year ago
    Running the
    PrefectSecret
    run
    method solves the
    TypeError
    . However, I can see in the console that it returns the value of SOME_SECRET.
    How is this different from passing in the secret as plaintext?
    Kevin Kho

    Kevin Kho

    1 year ago
    You’re right that it’s not. The PrefectSecret is really meant to be used in a Flow context. What are you using the secret for here that it needs to be passed as an env variable? Is it something that is needed to start the Flow run?
    You might be better off passing it to the agent using the
    --env
    flag?
    prefect agent kubernetes start --env SECRET=value
    ?
    m

    Michail Melonas

    1 year ago
    from prefect.engine.executors import DaskExecutor
    import coiled 
    
    executor = DaskExecutor(
        cluster_class=lambda: coiled.Cluster(**{
            "software": "somesoftwareenviron",
            "shutdown_on_close": False,
            "name": "prefect-executor",
            "n_workers": 1, 
            "credentials": "local",
        })
    )
    flow.executor = executor
    I'm trying to provision a Coiled cluster. I have a kubernetes agent sitting on a K8s cluster.
    Kevin Kho

    Kevin Kho

    1 year ago
    I’m not sure I’m following. Are you looking to pass the secret into
    credentials
    there?
    m

    Michail Melonas

    1 year ago
    No, the
    credentials
    argument is not the actual credentials, but is used to indicate which set of credentials to rely on In the docs:
    Kevin Kho

    Kevin Kho

    1 year ago
    Ah ok. Sorry, I’m not following how the Coiled cluster is related to the PrefectSecret?
    m

    Michail Melonas

    1 year ago
    In order to provision the Coiled cluster I need to authenticate using a token. I'd like to pass in the token as a
    PrefectSecret
    .
    Kevin Kho

    Kevin Kho

    1 year ago
    Ahh ok I understand. So I believe passing it through the RunConfig will have the secret exposed in some way. You can try adding it to your storage:
    flow.storage = S3(bucket="some-bucket", secrets=["AWS_CREDENTIALS", "SOME_SECRET"])
    and I think this will fetch it from Cloud when Prefect is loaded. Will try it in a bit.
    m

    Michail Melonas

    1 year ago
    Thanks, I'm gonna try this and get back to you.
    @Kevin Kho this did not solve my issue. I did, however, find a solution by adding an entrypoint to my Docker image that executes a Python script which fetches "SOME_SECRET" from Prefect Cloud using
    PrefectSecret("SOME_SECRET").run()
    .
    Kevin Kho

    Kevin Kho

    1 year ago
    Thanks for letting me know. I think you’re good but I’ll look into this more also.
    Chris L.

    Chris L.

    1 year ago
    @Michail Melonas you might have tried this already, but after you added
    secrets=["COILED_TOKEN"]
    to Storage, I'm wondering if you passed the secret from
    prefect.context.secrets
    into your custom function for creating the Coiled cluster: i.e.
    def coiled_cluster():
        import dask
        dask.config.set({"coiled.token": prefect.context.secrets.COILED_TOKEN})
        return coiled.Cluster(**{
            "software": "somesoftwareenviron",
            "shutdown_on_close": False,
            "name": "prefect-executor",
            "n_workers": 1,
        })
    
    flow.storage = S3(bucket="some-bucket", secrets=["COILED_TOKEN"])
    flow.executor = DaskExecutor(cluster_class=coiled_cluster)
    This pattern follows ideas from passing the "main image" (i.e. Docker image specified on run config for the Agent / flow runner) into the DaskExecutor (which manages the task runner) at runtime (see https://docs.prefect.io/orchestration/flow_config/executors.html#specifying-worker-images) I'm planning to test this pattern in the weekend, but wondering if you've tried this already (before you came to your Docker ENTRYPOINT solution)
    m

    Michail Melonas

    1 year ago
    Hey, @Chris L.. I have not tried this approach to passing in the secret. Let me know what you find
    Kevin Kho

    Kevin Kho

    1 year ago
    I believe this should work, just code the token might be exposed as plain text here
    Chris L.

    Chris L.

    1 year ago
    @Kevin Kho hi Kevin, could you kindly clarify where and when this code might be exposed? My assumption is that it shows up once on the agent as part of the process that starts up the executor. I don’t imagine the secret is saved to hard disk (or to any logs) on the agent after the flow run completes and the executor is torn down?
    Kevin Kho

    Kevin Kho

    1 year ago
    Not 100% sure here but my best answer is: It’s not saved to hard disk yep and you just have to be careful about not logging it. I think this is fine because it only runs on the client, but passing data to Dask workers in general does not encrypt stuff. So some people have concerns that this is being passed over the network