Questionnaire
05/19/2020, 4:26 PMJim Crist-Harif
05/19/2020, 4:29 PMQuestionnaire
05/19/2020, 4:30 PMJim Crist-Harif
05/19/2020, 4:35 PMQuestionnaire
05/19/2020, 4:40 PMJim Crist-Harif
05/19/2020, 4:45 PMAWS_CREDENTIALS
secret, as described here: https://docs.prefect.io/core/concepts/secrets.html#default-secrets
• Create a Secret("AWS_CREDENTIALS")
object as described here: https://docs.prefect.io/core/concepts/secrets.html#mechanisms
• Pass this secret as the credentials
parameter to the S3Upload
task as described here: https://docs.prefect.io/api/latest/tasks/aws.html#s3uploadQuestionnaire
05/19/2020, 4:55 PMAWS_CREDENTIALS
.... Do, I've to put it in my .bashrc
?Jim Crist-Harif
05/19/2020, 4:57 PMPREFECT__...
environment variable, or put it in the toml configuration in ~/.prefect/config.toml
. See https://docs.prefect.io/core/concepts/configuration.html for reference.Questionnaire
05/19/2020, 4:59 PMconfig.toml
will be automatically generated while installing prefect or I've to create it on my own?Jim Crist-Harif
05/19/2020, 5:00 PMQuestionnaire
05/19/2020, 5:00 PMTraceback (most recent call last):
File "main.py", line 20, in <module>
cred = Secret("AWS_CREDENTIALS").get()
File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 139, in get
) from None
ValueError: Local Secret "AWS_CREDENTIALS" was not found.
Jim Crist-Harif
05/19/2020, 5:11 PMQuestionnaire
05/19/2020, 5:12 PMconfig.toml
added
export PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS='{"ACCESS_KEY": "abcdef", "SECRET_ACCESS_KEY": "ghijklmn"}'
Jim Crist-Harif
05/19/2020, 5:19 PMPREFECT__
. There are many ways to set environment variables in linux, .bashrc
is one.
• Writing toml
in the config.toml
file.
In the above you've mixed the two. To set your config var in the toml file you'd write:
[context.secrets.AWS_CREDENTIALS]
ACCESS_KEY = "1234"
SECRET_ACCESS_KEY = "5678"
Questionnaire
05/19/2020, 5:26 PMTraceback (most recent call last):
File "main.py", line 20, in <module>
cred = Secret("AWS_CREDENTIALS").get()
File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 139, in get
) from None
ValueError: Local Secret "AWS_CREDENTIALS" was not found.
Jim Crist-Harif
05/19/2020, 5:30 PMfrom prefect import Flow, task
from prefect.tasks.secrets import PrefectSecret
@task
def print_secret(s):
print(s)
with Flow("test") as flow:
s = PrefectSecret("AWS_CREDENTIALS")
print_secret(s)
flow.run()
Questionnaire
05/19/2020, 5:30 PMJim Crist-Harif
05/19/2020, 5:31 PM~/.prefect/config.toml
of:
[context.secrets.AWS_CREDENTIALS]
ACCESS_KEY = "1234"
SECRET_ACCESS_KEY = "5678"
Questionnaire
05/19/2020, 5:36 PMSecret
instead of TaskSecret
.Jim Crist-Harif
05/19/2020, 5:42 PMQuestionnaire
05/19/2020, 5:46 PMJim Crist-Harif
05/19/2020, 5:47 PMQuestionnaire
05/19/2020, 5:51 PMJim Crist-Harif
05/19/2020, 5:53 PMQuestionnaire
05/19/2020, 6:01 PMJim Crist-Harif
05/19/2020, 8:53 PMcontext.secrets
config section is a valid secret. So the same steps as above, but replace AWS_CREDENTIALS
with whatever key you want.Questionnaire
05/19/2020, 9:15 PM[context.secrets.CONSTANTS]
BUCKET_NAME = "abcd"
Like if I put it like this and access using
a = Secret("CONSTANTS").get()
print(a)
It should work?Jim Crist-Harif
05/19/2020, 9:41 PMAWS_CREDENTIALS
secret above contained a dict. If you only want to map a key to a value you'd set:
[context.secrets]
KEY1 = "value1"
KEY2 = "value2"
to make PrefectSecret("KEY1")
and PrefectSecret("KEY2")
work.Questionnaire
05/19/2020, 9:49 PMrun
method of PrefectSecret
is giving errorJim Crist-Harif
05/19/2020, 10:12 PMQuestionnaire
05/19/2020, 10:15 PMTraceback (most recent call last):
File "main.py", line 23, in <module>
a = PrefectSecret("KEY1").run()
File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/utilities/tasks.py", line 279, in method
return run_method(self, *args, **kwargs)
File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/tasks/secrets/base.py", line 65, in run
return _Secret(name).get()
File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 143, in get
) from None
ValueError: Local Secret "KEY1" was not found.
Jim Crist-Harif
05/19/2020, 10:17 PMfrom prefect import Flow, task
from prefect.tasks.secrets import PrefectSecret
@task
def print_secret(s):
print(s)
with Flow("test") as flow:
s = PrefectSecret("EXAMPLE")
print_secret(s)
flow.run()
with the following config at ~/.prefect/config.toml
[context.secrets]
EXAMPLE = "hello"
KEY1
wasn't found in the prefect configuration, so you've set the secret incorrectly either in the toml file or as an environment variable.Questionnaire
05/19/2020, 10:23 PMTraceback (most recent call last):
File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/engine/task_runner.py", line 943, in get_task_run_state
self.task.run, timeout=self.task.timeout, **raw_inputs
File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/utilities/executors.py", line 182, in timeout_handler
return fn(*args, **kwargs)
File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/utilities/tasks.py", line 279, in method
return run_method(self, *args, **kwargs)
File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/tasks/secrets/base.py", line 65, in run
return _Secret(name).get()
File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 143, in get
) from None
ValueError: Local Secret "EXAMPLE" was not found.
Jim Crist-Harif
05/19/2020, 10:30 PM~/.prefect/config.toml
?Questionnaire
05/19/2020, 10:33 PMJim Crist-Harif
05/19/2020, 10:34 PMQuestionnaire
05/19/2020, 10:35 PM