https://prefect.io logo
Title
m

Michail Melonas

09/01/2021, 2:25 PM
I am having trouble passing sensitive information to my flow's run configuration. Having:
secret = PrefectSecret("SOME_SECRET")
flow.storage = S3(bucket="some-bucket", secrets=["AWS_CREDENTIALS"])

flow.run_config = KubernetesRun(
    image="<http://registry.gitlab.com/<some_image>|registry.gitlab.com/<some_image>>",
    labels=["prefect-14"],
    env={
        "PREFECT__LOGGING__LEVEL": "DEBUG",
        "SECRET": secret
    },
    image_pull_secrets=["image-puller"],
)
and then running:
flow.register("ProjectName")
I get:
TypeError: Object of type PrefectSecret is not JSON serializable
Any thoughts on this?
k

Kevin Kho

09/01/2021, 2:33 PM
Secrets need to be JSON Serializable. Is your secret a list or dict?
m

Michail Melonas

09/01/2021, 2:34 PM
It's a string saved on Prefect Cloud
k

Kevin Kho

09/01/2021, 2:37 PM
I think if it’s outside of the Flow, you need to do
PrefectSecret(…).run()
. Could you try that?
m

Michail Melonas

09/01/2021, 2:54 PM
Running the
PrefectSecret
run
method solves the
TypeError
. However, I can see in the console that it returns the value of SOME_SECRET.
How is this different from passing in the secret as plaintext?
k

Kevin Kho

09/01/2021, 3:00 PM
You’re right that it’s not. The PrefectSecret is really meant to be used in a Flow context. What are you using the secret for here that it needs to be passed as an env variable? Is it something that is needed to start the Flow run?
You might be better off passing it to the agent using the
--env
flag?
prefect agent kubernetes start --env SECRET=value
?
m

Michail Melonas

09/01/2021, 3:16 PM
from prefect.engine.executors import DaskExecutor
import coiled 

executor = DaskExecutor(
    cluster_class=lambda: coiled.Cluster(**{
        "software": "somesoftwareenviron",
        "shutdown_on_close": False,
        "name": "prefect-executor",
        "n_workers": 1, 
        "credentials": "local",
    })
)
flow.executor = executor
I'm trying to provision a Coiled cluster. I have a kubernetes agent sitting on a K8s cluster.
k

Kevin Kho

09/01/2021, 3:19 PM
I’m not sure I’m following. Are you looking to pass the secret into
credentials
there?
m

Michail Melonas

09/01/2021, 3:22 PM
No, the
credentials
argument is not the actual credentials, but is used to indicate which set of credentials to rely on In the docs:
k

Kevin Kho

09/01/2021, 3:26 PM
Ah ok. Sorry, I’m not following how the Coiled cluster is related to the PrefectSecret?
m

Michail Melonas

09/01/2021, 3:28 PM
In order to provision the Coiled cluster I need to authenticate using a token. I'd like to pass in the token as a
PrefectSecret
.
k

Kevin Kho

09/01/2021, 3:34 PM
Ahh ok I understand. So I believe passing it through the RunConfig will have the secret exposed in some way. You can try adding it to your storage:
flow.storage = S3(bucket="some-bucket", secrets=["AWS_CREDENTIALS", "SOME_SECRET"])
and I think this will fetch it from Cloud when Prefect is loaded. Will try it in a bit.
m

Michail Melonas

09/01/2021, 3:44 PM
Thanks, I'm gonna try this and get back to you.
@Kevin Kho this did not solve my issue. I did, however, find a solution by adding an entrypoint to my Docker image that executes a Python script which fetches "SOME_SECRET" from Prefect Cloud using
PrefectSecret("SOME_SECRET").run()
.
k

Kevin Kho

09/02/2021, 2:12 PM
Thanks for letting me know. I think you’re good but I’ll look into this more also.
c

Chris L.

09/15/2021, 2:03 AM
@Michail Melonas you might have tried this already, but after you added
secrets=["COILED_TOKEN"]
to Storage, I'm wondering if you passed the secret from
prefect.context.secrets
into your custom function for creating the Coiled cluster: i.e.
def coiled_cluster():
    import dask
    dask.config.set({"coiled.token": prefect.context.secrets.COILED_TOKEN})
    return coiled.Cluster(**{
        "software": "somesoftwareenviron",
        "shutdown_on_close": False,
        "name": "prefect-executor",
        "n_workers": 1,
    })

flow.storage = S3(bucket="some-bucket", secrets=["COILED_TOKEN"])
flow.executor = DaskExecutor(cluster_class=coiled_cluster)
This pattern follows ideas from passing the "main image" (i.e. Docker image specified on run config for the Agent / flow runner) into the DaskExecutor (which manages the task runner) at runtime (see https://docs.prefect.io/orchestration/flow_config/executors.html#specifying-worker-images) I'm planning to test this pattern in the weekend, but wondering if you've tried this already (before you came to your Docker ENTRYPOINT solution)
m

Michail Melonas

09/15/2021, 6:42 AM
Hey, @Chris L.. I have not tried this approach to passing in the secret. Let me know what you find
k

Kevin Kho

09/15/2021, 1:56 PM
I believe this should work, just code the token might be exposed as plain text here
c

Chris L.

09/15/2021, 3:36 PM
@Kevin Kho hi Kevin, could you kindly clarify where and when this code might be exposed? My assumption is that it shows up once on the agent as part of the process that starts up the executor. I don’t imagine the secret is saved to hard disk (or to any logs) on the agent after the flow run completes and the executor is torn down?
k

Kevin Kho

09/15/2021, 4:06 PM
Not 100% sure here but my best answer is: It’s not saved to hard disk yep and you just have to be careful about not logging it. I think this is fine because it only runs on the client, but passing data to Dask workers in general does not encrypt stuff. So some people have concerns that this is being passed over the network