https://prefect.io logo
Join Slack
Powered by
My understanding is that the Agent deployment is w...
# ask-community
w
My understanding is that the Agent deployment is what needs to have local secrets configured, in order to use for example GitHub storage?
I’ve got:
Copy code
- name: PREFECT__CONTEXT__SECRETS__GITHUB_TOKEN
  value: "redacted"
in the agent deployment env config, but no secret named ‘github_token’ seems to be available when flows run
k
Maybe try setting 
"PREFECT__CLOUD__USE_LOCAL_SECRETS" = "true"
w
Sadly that doesn’t seem to do it; I tried setting that on the Agent as well as on the worker spec
k
Can I see the error message?
w
Let me get back to it; I tried adding a workaround, and that gave me: 
ValueError: Secrets should only be retrieved during a Flow run, not while building a Flow.
But I need the Secret for the GitHub storage config
k
So you did something like
Copy code
my_secret_value = Secret("MYSECRET").get()
outside the flow?
w
Yeah
Copy code
+ exec prefect execute flow-run
[2021-08-30 01:01:06+0000] INFO - prefect.GitHub | Downloading flow from GitHub storage - repo: 'Addepar/comparch-ng', path: 'addemart/src/addemart/__main__.py', ref: 'wlb/prefect-demo'
Local Secret "github_token" was not found.
Traceback (most recent call last):
  File "/opt/conda/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/prefect/cli/execute.py", line 96, in flow_run
    raise exc
  File "/opt/conda/lib/python3.8/site-packages/prefect/cli/execute.py", line 73, in flow_run
    flow = storage.get_flow(flow_data.name)
  File "/opt/conda/lib/python3.8/site-packages/prefect/storage/github.py", line 94, in get_flow
    client = self._get_github_client()
  File "/opt/conda/lib/python3.8/site-packages/prefect/storage/github.py", line 161, in _get_github_client
    access_token = Secret(self.access_token_secret).get()
  File "/opt/conda/lib/python3.8/site-packages/prefect/client/secrets.py", line 140, in get
    raise ValueError(
ValueError: Local Secret "github_token" was not found.
I added this just to see what would happen, and got the error about doing it outside a flow:
Copy code
client = prefect.Client()
    token = os.environ.get("PREFECT__CONTEXT__SECRETS__GITHUB_TOKEN", Secret("github_token").get())
    client.set_secret(name="github_token", value=token)
k
I am wondering if this behavior is related to this.
Actually ignore that. That’s specifically for the API
Ok I’m positive this time. This is what you’re running into
w
Hmm, I see. So I’ve only exposed the env var to the agent process, not configured the agent to spawn processes with that env added
👍 1
Grr, bummer, same error even after adding 
--env PREFECT_FOO_BAZ="value"
to the agent startup command
I was expecting that to work
k
At this point I would log the context in the prefect flow to see what is getting populated cuz that should really work
w
The flow can’t get fetched from GitHub though because it’s missing this secret. Hmm.
It’s weird because I have the env vars in the spec for the worker pods as well
Copy code
def _worker_env() -> Dict[str, str]:
    env = {
        "EXTRA_PIP_PACKAGES": "prefect[kubernetes,github] dask-kubernetes",
        "MYSQL_HOST": "redacted",
        "ON_AWS": "1",
        "POSTGRES_HOST": "redacted",
        "PREFECT__CLOUD__USE_LOCAL_SECRETS": "true",
        "PREFECT__CONTEXT__SECRETS__GITHUB_TOKEN": "redacted",
        "TZ": "UTC",
    }
    return env
I set that as ‘env’ on both the Executor and RunConfig
k
Ah that’s true. The agent with 
--env PREFECT__CONTEXT__SECRETS__GITHUB_TOKEN=xxxx
should work. I can give it a try later on a Kubernetes agent.
From here . So maybe you need to capitalize when you fetch the secret
w
OMG that seems to be it; it is case-sensitive
I’m getting repo-not-found but that’s certainly me just having created the token incorrectly or something
Ok, I am super close, but I have another dumb question..
Copy code
+ exec prefect execute flow-run
[2021-08-30 03:36:31+0000] INFO - prefect.GitHub | Downloading flow from GitHub storage - repo: 'Addepar/comparch-ng', path: 'addemart/src/addemart/__main__.py', ref: 'wlb/prefect-demo'
[2021-08-30 03:36:31+0000] INFO - prefect.GitHub | Flow successfully downloaded. Using commit: 1823a38bf6e1140a4fe10173e39e4a30363a8ce4
No module named 'addemart'
Traceback (most recent call last):
  File "/opt/conda/bin/prefect", line 8, in <module>
    sys.exit(cli())
 ...
  File "/opt/conda/lib/python3.8/site-packages/prefect/storage/github.py", line 128, in get_flow
    return extract_flow_from_file(
  File "/opt/conda/lib/python3.8/site-packages/prefect/utilities/storage.py", line 88, in extract_flow_from_file
    exec(contents, exec_vals)
  File "<string>", line 31, in <module>
ModuleNotFoundError: No module named 'addemart'
addemart
is the name of the module that registers and runs the flows. Normally it would be `pip install`’d from the checkout dir.
Is there a way to run a setup step like that on the agent->worker, when they download the code?
k
Do you mean you want to install 
addemart
on the Dask workers? How do you download it?
w
It’s already there in the GitHub repo that the flow fetches
In local dev we do that via: 
pip install --editable ${CURDIR}[dev,test]
but just 
pip install .
would do it on a worker I think
We have a valid 
setup.py
k
The 
GitHub
storage only pulls the Flow file and not other files that get imported. You might need Docker storage to hold that dependency and to 
pip install .
when building the Docker container. 
Git
storage pulls the whole repo but it is not intended to be 
pip installed
like that. It’s more for holding 
.sql
files or 
.yaml
files your flow may use. I think the flow is running Python process already and you’d need to restart it to have the 
pip install
take effect right?
w
Hmm. I didn't realize that; I figured it was a bare clone or something. I see.
It's brutal how close to a final implementation I'm having to come just to get my demo working 😕
So, introducing Docker into this.. which actor actually pulls the image, and needs 
pullSecrets
? The Agent? Or is that Dask's KubeCluster handling that?
k
I know 😞 . I would say the Dask KubeCluster because that’s the one that loads the images. I would say that you can stay with Github storage, and then just change the image for Dask
14 Views
Open in Slack
PreviousNext
Powered by