Mars
05/03/2022, 2:29 PMprefect k8s agent install
. I’ve uploaded my flow to a private GitHub repo and registered it with Prefect. And I’ve added a Cloud Secret called GITHUB_ACCESS_TOKEN
that holds a valid GitHub personal access token. When I run my flow the agent’s GitHub storage gives me an UnknownObjectException(404, 'Not Found')
error. If I change the flow to use a different Cloud Secret key for the PAT, such as access_token_secret='MYKEY'
, then the agent tells me ValueError('Local Secret "MYKEY" was not found.')
.
How can I introspect the kubernetes agent to verify that the GitHub PAT secret is being loaded from Prefect Cloud correctly?Kevin Kho
05/03/2022, 2:34 PMPREFECT___CLOUD____USE__LOCAL_SECRETS
set to True? 404 is the repo couldn’t be found I think. Could you show me what your Github storage looks like?Mars
05/03/2022, 2:38 PMPREFECT___CLOUD____USE__LOCAL_SECRETS
key. It’s set to whatever the default value is. I’m looking at the agent logs in the pod but there isn’t much there: it mentions the labels but doesn’t display any startup configuration.Kevin Kho
05/03/2022, 2:40 PMGITHUB_ACCESS_TOKEN
seeme fineMars
05/03/2022, 2:40 PMKevin Kho
05/03/2022, 2:40 PMMars
05/03/2022, 2:42 PMstorage = GitHub(
repo="myorg/sandbox",
path="flows/check_repo_access.py",
)
# Taken from the 'script-based flows' example
with Flow("check-repo-access", storage=storage) as flow:
data = get_data()
print_value(data)
Kevin Kho
05/03/2022, 2:44 PMMars
05/03/2022, 2:44 PMstorage = GitHub(
repo="myorg/sandbox",
path="flows/check_repo_access.py",
access_token_secret="GITHUB_ACCESS_TOKEN",
)
or this:
storage = GitHub(
repo="myorg/sandbox",
path="flows/check_repo_access.py",
access_token_secret="MYKEY",
)
Kevin Kho
05/03/2022, 2:49 PMwith Flow(..) as flow:
...
storage =...
flow.storage = ...
storage.get_flow(flow_name)
and this get_flow
is what the agent calls.
For agent settings, you can add
--show-flow-logs
and
--log-level=DEBUG
upon agent startMars
05/03/2022, 3:17 PMFOO
. When I run that flow the agent gives me a pickle error about the flow running Python 3.7.13. However I deployed the agent image prefecthq/prefect:1.2.0-python3.9
and I verified that the Python version in the agent is 3.9.12. Looks like a bug with the flow execution environment?
import prefect
from prefect import Flow, task
from prefect.run_configs import UniversalRun
from prefect.storage import S3
from prefect.tasks.secrets import PrefectSecret
@task
def print_value(secret):
logger = prefect.context.get("logger")
<http://logger.info|logger.info>(f"value: {secret}")
with Flow(
"print-cloud-secret",
storage=S3(bucket="my-bucket"),
run_config=UniversalRun(labels=[])) \
as flow:
s = PrefectSecret("FOO")
print_value(s)
if __name__ == "__main__":
flow.run()
└── 11:06:54 | INFO | Entered state <Failed>: Failed to load and execute flow run: FlowStorageError("An error occurred while unpickling the flow:\n TypeError('code() takes at most 15 arguments (16 given)')\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - python: (flow built with '3.10.4', currently running with '3.7.13')")
└── 11:06:54 | ERROR | Failed to load and execute flow run: FlowStorageError("An error occurred while unpickling the flow:\n TypeError('code() takes at most 15 arguments (16 given)')\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - python: (flow built with '3.10.4', currently running with '3.7.13')")
Kevin Kho
05/03/2022, 3:19 PMMars
05/03/2022, 3:24 PMKevin Kho
05/03/2022, 3:24 PMprefecthq/prefect:latest
which is 3.7 by default yep 🙂. You’d need to choose a differently tagged image (I am not sure we support 3.10)Mars
05/03/2022, 3:30 PMValueError: Local Secret "FOO" was not found.
That means that the call to PrefectSecret("FOO")
is not pulling values from the cloud vault. Why would that be?FOO
is a valid key under https://cloud.prefect.io/team/secretsKevin Kho
05/03/2022, 3:31 PMflow.run()
somehow? Because flow.run()
will attempt to pull locallyMars
05/03/2022, 3:31 PMKevin Kho
05/03/2022, 3:32 PMconfig.toml
somewhere?Mars
05/03/2022, 3:32 PMflow.run()
except in a if name == main
block:
with Flow(
"print-cloud-secret",
storage=S3(bucket="my-bucket"),
run_config=UniversalRun(labels=[])) \
as flow:
s = PrefectSecret("FOO")
print_value(s)
if __name__ == "__main__":
flow.run()
Kevin Kho
05/03/2022, 3:32 PMPREFECT__CLOUD__USE_LOCAL_SECRETS=false
Mars
05/03/2022, 3:33 PMKevin Kho
05/03/2022, 3:33 PMwith Flow(
"print-cloud-secret",
storage=S3(bucket="my-bucket"),
run_config=KubernetesRun(image="...", labels=[])) \
as flow:
s = PrefectSecret("FOO")
print_value(s)
You are on Kubernetes right?Mars
05/03/2022, 3:34 PMimport prefect
from prefect import Flow, task
from prefect.run_configs import UniversalRun, KubernetesRun
from prefect.storage import S3
from prefect.tasks.secrets import PrefectSecret
@task
def print_value(secret):
logger = prefect.context.get("logger")
<http://logger.info|logger.info>(f"value: {secret}")
with Flow(
"print-cloud-secret",
storage=S3(bucket="my-bucket"),
run_config=KubernetesRun(
image="prefecthq/prefect:1.2.0-python3.9",
image_pull_policy="IfNotPresent",
labels=[])) \
as flow:
s = PrefectSecret("FOO")
print_value(s)
if __name__ == "__main__":
flow.run()
Kevin Kho
05/03/2022, 3:37 PMwith Flow(
"print-cloud-secret",
storage=S3(bucket="my-bucket"),
run_config=KubernetesRun(
image="prefecthq/prefect:1.2.0-python3.9",
image_pull_policy="IfNotPresent",
env={"PREFECT__CLOUD__USE_LOCAL_SECRETS": False}
labels=[])) \
as flow:
s = PrefectSecret("FOO")
print_value(s)
Mars
05/03/2022, 3:38 PMprefect run --name print-cloud-secret --watch
I verified that the job was run by the agent by checking the agent logs. I can see the failed run on the Kubernetes Agent dashboard in the Cloud UI.env={"PREFECT__CLOUD__USE_LOCAL_SECRETS": False}
from the run_config. Still fails saying the local secret was not found.Kevin Kho
05/03/2022, 3:42 PMMars
05/03/2022, 3:51 PMKevin Kho
05/03/2022, 3:56 PMfrom prefect.client import Secret
Secret("GITHUB_ACCESS_TOKEN").get()
inside the pod?Mars
05/03/2022, 3:56 PMroot@prefect-agent-69688996d-qltbc:/# python
Python 3.9.12 (main, Mar 29 2022, 14:20:48)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from prefect.client import Secret
>>> Secret("GITHUB_ACCESS_TOKEN").get()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 140, in get
raise ValueError(
ValueError: Local Secret "GITHUB_ACCESS_TOKEN" was not found.
Kevin Kho
05/03/2022, 4:02 PMimport os
os.environ["PREFECT__CLOUD__USE_LOCAL_SECRETS"] = "false"
from prefect.client import Secret
Secret("GITHUB_ACCESS_TOKEN").get()
Mars
05/03/2022, 4:04 PM>>> import os
>>> os.environ["PREFECT__CLOUD__USE_LOCAL_SECRETS"] = "false"
>>>
>>> from prefect.client import Secret
>>> Secret("GITHUB_ACCESS_TOKEN").get()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 140, in get
raise ValueError(
ValueError: Local Secret "GITHUB_ACCESS_TOKEN" was not found.
Kevin Kho
05/03/2022, 4:04 PMKeyError: 'The secret SECRET_NAME was not found. Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>'
It should only go there if you are not configured to hit Cloud as a backendimport os
os.environ["PREFECT__CLOUD__USE_LOCAL_SECRETS"] = "false"
os.environ["PREFECT__BACKEND"] = "cloud"
from prefect.client import Secret
print(Secret("GCP_CREDENTIALS").get())
Mars
05/03/2022, 4:14 PM>>> os.environ.get("PREFECT__BACKEND")
'server'
>>>
Kevin Kho
05/03/2022, 4:16 PMMars
05/03/2022, 4:16 PM>>> os.environ.get("PREFECT__BACKEND")
'server'
>>> import os
>>> os.environ["PREFECT__CLOUD__USE_LOCAL_SECRETS"] = "false"
>>> os.environ["PREFECT__BACKEND"] = "cloud"
>>>
>>> from prefect.client import Secret
>>> print(Secret("GCP_CREDENTIALS").get())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 140, in get
raise ValueError(
ValueError: Local Secret "GCP_CREDENTIALS" was not found.
Kevin Kho
05/03/2022, 4:16 PMMars
05/03/2022, 4:17 PMprefect run
that I can verify with?Kevin Kho
05/03/2022, 4:17 PMMars
05/03/2022, 4:18 PMKevin Kho
05/03/2022, 4:22 PMPREFECT__BACKEND="server"
PREFECT__SERVER__ENDPOINT="<http://api.prefect.io|api.prefect.io>"
which is why it works.Mars
05/03/2022, 5:06 PMprefect agent kubernetes install
before I had run prefect backend cloud
to switch it over. That server
setting ended up in the manifest that I deployed the agent with. I switched the backend on the CLI later but I didn’t regenerate the manifest from scratch.Kevin Kho
05/03/2022, 5:08 PMMars
05/03/2022, 5:14 PMserver
agent is in the Cloud UI? A big red warning label to say your agent is probably misconfigured
would have saved a lot of time.server
backend and <http://cloud.prefect.io|cloud.prefect.io>
API URL then something is probably wrongKevin Kho
05/03/2022, 5:16 PM