https://prefect.io logo
j

Jons Cyriac

01/25/2022, 1:23 PM
Can context secrets be passed from agent to dask executors during flow run? Sorry if im repeating this question. Im trying to get some credentials available on the dask workers, I have set envs like
PREFECT__CONTEXT__SECRETS__DB_URI
on the agent. Is this supposed to be available as prefect.context.secrets["DB_URI"] when the flow is run on dask worker without any other setup?
this is how my flow config looks like:
a

Anna Geller

01/25/2022, 1:48 PM
Are you on Prefect Cloud or Prefect Server?
j

Jons Cyriac

01/25/2022, 1:52 PM
prefect server
a

Anna Geller

01/25/2022, 2:02 PM
Thanks. In that case you indeed have to use local secrets. Generally, local secrets are retrieved from the agent environment so if you set it within the agent (e.g. when starting it), then the secret should be available within your task (regardless of whether the task run gets executed locally or on Dask). Example:
Copy code
$ prefect agent docker start --env PREFECT__CONTEXT__SECRETS__CRED="CRED_VALUE"
And if normal PrefectSecret doesn’t work, you may try the `EnvVarSecret`:
Copy code
from prefect import task, Flow
from prefect.tasks.secrets import EnvVarSecret

@task(log_stdout=True)
def print_value(x):
    print(x)

with Flow("Example") as flow:
    secret = EnvVarSecret("CRED")
    print_value(secret)

flow.run() # prints the value of the "CRED" environment variable
But if this doesn’t work, you can set it directly in your config.toml on the agent:
Copy code
[context.secrets]
MYSECRET = "MY SECRET VALUE"
j

Jons Cyriac

01/25/2022, 2:13 PM
im using prefect's helm chart to deploy this. I dont think (know how) I'll be able to pass --env to agent start command directly. How and where do I set value for CRED if I'm using
EnvVarSecret
?
a

Anna Geller

01/25/2022, 2:15 PM
You would put it in the env variables section:
Copy code
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect-agent
  name: prefect-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-agent
  template:
    metadata:
      labels:
        app: prefect-agent
    spec:
      containers:
      - args:
        - prefect agent kubernetes start
        command:
        - /bin/bash
        - -c
        env:
        - name: PREFECT__CONTEXT__SECRETS__CRED
          value: 'CRED_VALUE'
j

Jons Cyriac

01/25/2022, 2:34 PM
great, this makes CRED available to tasks using prefect.context.secrets.CRED OR EnvVarSecret("CRED")?
a

Anna Geller

01/25/2022, 2:34 PM
it should, yes
j

Jons Cyriac

01/25/2022, 2:40 PM
cool, thanks Anna
Looks like EnvVarSecret is able to pick it up
👍 1
@Anna Geller How do I specify the dask cluster address to DaskExeceutor at run time? i dont want to specify it at flow registration.
a

Anna Geller

01/25/2022, 3:15 PM
the executor is retrieved from storage (rather than from the backend), so you don’t need to provide it at registration. As long as your storage contains the Dask address, this should work
j

Jons Cyriac

01/25/2022, 3:21 PM
so post registration of a flow, I have to modify the flow in the storage if theres a change in the executor URI? 😕 how do I go about that?
a

Anna Geller

01/25/2022, 3:36 PM
if your Dask address changes, you can just update this in your flow storage. You can e.g. use script storage and just upload your changed file to your GCS bucket - you could even do it as part of your CI/CD pipeline
j

Jons Cyriac

01/25/2022, 3:42 PM
okay, one more question that I have is, whats the right way for create_flow_run to refer to a particular flow from a list of many flows? I want to trigger certain flows from apis. Any way to trigger it by name? does prefect Client support listing flows?
a

Anna Geller

01/25/2022, 3:45 PM
The create_flow_run task does support triggering a flow run by flow name already. To list specific flows and their names, you can use the GraphQL API, e.g.
Copy code
query {
  flow(where: { name: { _ilike: "%dask%" } }) {
    id
    name
    version
  }
}
🙌 1
j

Jons Cyriac

01/25/2022, 3:47 PM
thanks
👍 1
Sorry Anna, the EnvVarSecret is not working
this is my value.yaml for the helm-chart for agent
context.secret is populated only if I manually edit the deployment and add --env to the start command. (highlighted text in this image)
a

Anna Geller

01/25/2022, 5:52 PM
I see. If setting this on the agent startup command works, I would use that, there is nothing that speaks against it imo. But if you feel like it’s a deal breaker for you, feel free to open an issue in the Server repository and we could investigate more.
📝 1
5 Views