https://prefect.io logo
Powered by
Title
q

Questionnaire

05/19/2020, 4:26 PM
Hey folks, I'm a begginer here.... Can someone share me the step by step process for third party authentication like AWS. I read docs but it's not clear to me.
j

Jim Crist-Harif

05/19/2020, 4:29 PM
Hi Tanmay. Can you clarify what you're trying to do? Are you trying to run a task that uses some AWS apis? Or are you trying to run a flow on AWS?
q

Questionnaire

05/19/2020, 4:30 PM
I want to use S3 upload in a task
j

Jim Crist-Harif

05/19/2020, 4:35 PM
The simplest possible thing would be to get boto3 setup working locally in your environment, and avoid working with prefect's secrets at all. Once you get that working you may transition to working with secrets if needed, but starting simple is a good place.
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration
q

Questionnaire

05/19/2020, 4:40 PM
Ohk, thanks but I want to do it using secrets... Can you walk me through it?
j

Jim Crist-Harif

05/19/2020, 4:45 PM
Sure. You'd need to: • Configure the 
AWS_CREDENTIALS
secret, as described here: https://docs.prefect.io/core/concepts/secrets.html#default-secrets • Create a 
Secret("AWS_CREDENTIALS")
object as described here: https://docs.prefect.io/core/concepts/secrets.html#mechanisms • Pass this secret as the 
credentials
parameter to the 
S3Upload
task as described here: https://docs.prefect.io/api/latest/tasks/aws.html#s3upload
q

Questionnaire

05/19/2020, 4:55 PM
Hey @Jim Crist-Harif, I'm using Ubuntu while Configuring 
AWS_CREDENTIALS
.... Do, I've to put it in my 
.bashrc
?
j

Jim Crist-Harif

05/19/2020, 4:57 PM
This is using prefect's configuration mechanism. So you could set the 
PREFECT__...
environment variable, or put it in the toml configuration in 
~/.prefect/config.toml
. See https://docs.prefect.io/core/concepts/configuration.html for reference.
q

Questionnaire

05/19/2020, 4:59 PM
config.toml
will be automatically generated while installing prefect or I've to create it on my own?
j

Jim Crist-Harif

05/19/2020, 5:00 PM
You'll need to create it on your own.
q

Questionnaire

05/19/2020, 5:00 PM
Ohh, ohk let me try once
I did exactly what is there in docs and link you shared. But I'm getting this error 
Traceback (most recent call last):
  File "main.py", line 20, in <module>
    cred = Secret("AWS_CREDENTIALS").get()
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 139, in get
    ) from None
ValueError: Local Secret "AWS_CREDENTIALS" was not found.
j

Jim Crist-Harif

05/19/2020, 5:11 PM
How did you configure the secret?
q

Questionnaire

05/19/2020, 5:12 PM
I created the file 
config.toml
added 
export PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS='{"ACCESS_KEY": "abcdef", "SECRET_ACCESS_KEY": "ghijklmn"}'
and trying to acccess the secret using https://docs.prefect.io/core/concepts/secrets.html#mechanisms
j

Jim Crist-Harif

05/19/2020, 5:19 PM
Ah, yeah. So you have two options for configuring prefect: • Setting keys using environment variables starting with 
PREFECT__
. There are many ways to set environment variables in linux, 
.bashrc
is one. • Writing 
toml
in the 
config.toml
file. In the above you've mixed the two. To set your config var in the toml file you'd write: 
[context.secrets.AWS_CREDENTIALS]
ACCESS_KEY = "1234"
SECRET_ACCESS_KEY = "5678"
q

Questionnaire

05/19/2020, 5:26 PM
I changed my .toml to exactly what you suggested but I'm getting the same error.
Traceback (most recent call last):
  File "main.py", line 20, in <module>
    cred = Secret("AWS_CREDENTIALS").get()
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 139, in get
    ) from None
ValueError: Local Secret "AWS_CREDENTIALS" was not found.
j

Jim Crist-Harif

05/19/2020, 5:30 PM
Where did you put your config file?
The following works fine for me: 
from prefect import Flow, task
from prefect.tasks.secrets import PrefectSecret

@task
def print_secret(s):
    print(s)


with Flow("test") as flow:
    s = PrefectSecret("AWS_CREDENTIALS")
    print_secret(s)


flow.run()
q

Questionnaire

05/19/2020, 5:30 PM
Ok let me try
j

Jim Crist-Harif

05/19/2020, 5:31 PM
with 
~/.prefect/config.toml
of: 
[context.secrets.AWS_CREDENTIALS]
ACCESS_KEY = "1234"
SECRET_ACCESS_KEY = "5678"
q

Questionnaire

05/19/2020, 5:36 PM
Let me tell you the issue, I was using 
Secret
instead of 
TaskSecret
.
But now I'm facing another issue where it's.... TypeError: run() got an unexpected keyword argument 'credentials'
in S3Upload
j

Jim Crist-Harif

05/19/2020, 5:42 PM
What version of prefect are you using? That keyword argument does exist.
q

Questionnaire

05/19/2020, 5:46 PM
Oh, I was using 0.10.7
so there they've ``aws_credentials_secret``
j

Jim Crist-Harif

05/19/2020, 5:47 PM
Yeah, providing as a runtime parameter is new in 0.11.0
if possible I recommend upgrading.
q

Questionnaire

05/19/2020, 5:51 PM
Yeah, I did it already.
Thank you so much @Jim Crist-Harif. You're Awesome, would like to connect you on LinkedIn: (https://www.linkedin.com/in/tanmay-srivastava-1a9279129/)
j

Jim Crist-Harif

05/19/2020, 5:53 PM
Glad to help, hope prefect is useful for you. And I don't have a linkedin :).
q

Questionnaire

05/19/2020, 6:01 PM
Ohk No issues... Thanks man 🙂
Hey @Jim Crist-Harif can you tell me how to set custom secret?
j

Jim Crist-Harif

05/19/2020, 8:53 PM
Any field in the 
context.secrets
config section is a valid secret. So the same steps as above, but replace 
AWS_CREDENTIALS
with whatever key you want.
q

Questionnaire

05/19/2020, 9:15 PM
[context.secrets.CONSTANTS]
BUCKET_NAME = "abcd"
Like if I put it like this and access using 
a =  Secret("CONSTANTS").get()
print(a)
It should work?
j

Jim Crist-Harif

05/19/2020, 9:41 PM
Sorry, the 
AWS_CREDENTIALS
secret above contained a dict. If you only want to map a key to a value you'd set: 
[context.secrets]
KEY1 = "value1"
KEY2 = "value2"
to make 
PrefectSecret("KEY1")
and 
PrefectSecret("KEY2")
work.
q

Questionnaire

05/19/2020, 9:49 PM
Ohhh ohkkk
run
method of 
PrefectSecret
is giving error
j

Jim Crist-Harif

05/19/2020, 10:12 PM
You'll need to post the error for me to help debug.
q

Questionnaire

05/19/2020, 10:15 PM
Traceback (most recent call last):
  File "main.py", line 23, in <module>
    a =  PrefectSecret("KEY1").run()
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/utilities/tasks.py", line 279, in method
    return run_method(self, *args, **kwargs)
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/tasks/secrets/base.py", line 65, in run
    return _Secret(name).get()
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 143, in get
    ) from None
ValueError: Local Secret "KEY1" was not found.
j

Jim Crist-Harif

05/19/2020, 10:17 PM
The following works for me: 
from prefect import Flow, task
from prefect.tasks.secrets import PrefectSecret

@task
def print_secret(s):
    print(s)


with Flow("test") as flow:
    s = PrefectSecret("EXAMPLE")
    print_secret(s)


flow.run()
with the following config at 
~/.prefect/config.toml
[context.secrets]
EXAMPLE = "hello"
The error you're seeing indicates that the secret key 
KEY1
wasn't found in the prefect configuration, so you've set the secret incorrectly either in the toml file or as an environment variable.
q

Questionnaire

05/19/2020, 10:23 PM
Traceback (most recent call last):
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/engine/task_runner.py", line 943, in get_task_run_state
    self.task.run, timeout=self.task.timeout, **raw_inputs
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/utilities/executors.py", line 182, in timeout_handler
    return fn(*args, **kwargs)
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/utilities/tasks.py", line 279, in method
    return run_method(self, *args, **kwargs)
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/tasks/secrets/base.py", line 65, in run
    return _Secret(name).get()
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 143, in get
    ) from None
ValueError: Local Secret "EXAMPLE" was not found.
j

Jim Crist-Harif

05/19/2020, 10:30 PM
And your config looks like the one I posted and is stored in the proper location at 
~/.prefect/config.toml
?
q

Questionnaire

05/19/2020, 10:33 PM
It was a issue with the format, I had AWS_CREDENTIALS secret also in the same file, so formatting got incorrect
j

Jim Crist-Harif

05/19/2020, 10:34 PM
glad you figured it out
q

Questionnaire

05/19/2020, 10:35 PM
Thanks man you're a savior.
2 Views
#prefect-communityJoin Slack
Powered by