bruno.corucho
07/10/2020, 1:51 PMCOPY configuration /opt/strdata-prefect/configuration
ENV PREFECT__USER_CONFIG_PATH /opt/strdata-prefect/configuration/config.toml
Config.toml:
backend = "cloud"
[cloud]
use_local_secrets = false
[cloud.agent]
name = "strdata-agent"
# Setting it to `DEBUG` for verbose logging
level = "DEBUG"
[logging]
# The logging level: NOTSET, DEBUG, INFO, WARNING, ERROR, or CRITICAL
level = "DEBUG"
# Send logs to Prefect Cloud
log_to_cloud = true
# Extra loggers for Prefect log configuration
extra_loggers = "[]"
Error Logs:
ValueError: Local Secret "REDSHIFT_PASSWORD" was not found.
Thanks again, Prefect Team!nicholas
bruno.corucho
07/10/2020, 1:54 PMnicholas
config.toml
at ~/.prefect
?bruno.corucho
07/10/2020, 2:00 PMnicholas
printenv
should confirm this), try moving your config to the normal location and see if that worksLuis Muniz
07/10/2020, 3:28 PMValueError Local Secret "REDSHIFT_PASSWORD" was not found.
And we solved it by adding this directive:
ENV PREFECT__CONTEXT__SECRETS__REDSHIFT_PASSWORD "unused"
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
File "/opt/prefect/healthcheck.py", line 117, in <module>
flows = cloudpickle_deserialization_check(flow_file_path)
File "/opt/prefect/healthcheck.py", line 39, in cloudpickle_deserialization_check
flows.append(cloudpickle.load(f))
File "/opt/strdata-prefect/strdata/twitch_tasks/collect_streamers.py", line 19, in <module>
API_CLIENT_ID = Secret("API_CLIENT_ID").get() # os.environ["API_CLIENT_ID"]
File "/usr/local/lib/python3.8/site-packages/prefect/client/secrets.py", line 163, in get
raise ValueError(
ValueError: Local Secret "API_CLIENT_ID" was not found.
nicholas
Luis Muniz
07/10/2020, 4:15 PMnicholas
Luis Muniz
07/10/2020, 5:19 PMREDSHIFT_PASSWORD = Secret("REDSHIFT_PASSWORD").get()
#now defining a tesk
@task
def dosomething():
connect_to_redshift(REDSHIFT_PASSWORD)
nicholas
Secrets
are tasks, you're essentially trying to execute the task outside of a Prefect context. My suggestion is to move your secrets into the Flow context like this:
@task
def i_need_redshift(redshift_password):
password = redshift_password.get()
# do something with password
with Flow("your_flow") as flow:
redshift_password = Secret("REDSHIFT_PASSWORD")
i_need_redshift(redshift_password)
or if you'd prefer not to build the task into your dependency tree that way, you can call it inside the tasks that need it instead:
@task
def i_need_redshift(redshift_password):
redshift_password = Secret("REDSHIFT_PASSWORD").get()
# do something with redshift_password
Luis Muniz
07/10/2020, 5:27 PMbruno.corucho
07/10/2020, 5:27 PMLuis Muniz
07/10/2020, 5:27 PMnicholas
bruno.corucho
07/10/2020, 5:29 PMnicholas
Luis Muniz
07/10/2020, 5:30 PM