https://prefect.io logo
Join Slack
Powered by
Hey folks, I'm a begginer here.... Can someone sha...
# prefect-community
q
Hey folks, I'm a begginer here.... Can someone share me the step by step process for third party authentication like AWS. I read docs but it's not clear to me.
j
Hi Tanmay. Can you clarify what you're trying to do? Are you trying to run a task that uses some AWS apis? Or are you trying to run a flow on AWS?
q
I want to use S3 upload in a task
j
The simplest possible thing would be to get boto3 setup working locally in your environment, and avoid working with prefect's secrets at all. Once you get that working you may transition to working with secrets if needed, but starting simple is a good place.
q
Ohk, thanks but I want to do it using secrets... Can you walk me through it?
j
Sure. You'd need to: • Configure the 
AWS_CREDENTIALS
secret, as described here: https://docs.prefect.io/core/concepts/secrets.html#default-secrets • Create a 
Secret("AWS_CREDENTIALS")
object as described here: https://docs.prefect.io/core/concepts/secrets.html#mechanisms • Pass this secret as the 
credentials
parameter to the 
S3Upload
task as described here: https://docs.prefect.io/api/latest/tasks/aws.html#s3upload
q
Hey @Jim Crist-Harif, I'm using Ubuntu while Configuring 
AWS_CREDENTIALS
.... Do, I've to put it in my 
.bashrc
?
j
This is using prefect's configuration mechanism. So you could set the 
PREFECT__...
environment variable, or put it in the toml configuration in 
~/.prefect/config.toml
. See https://docs.prefect.io/core/concepts/configuration.html for reference.
q
config.toml
will be automatically generated while installing prefect or I've to create it on my own?
j
You'll need to create it on your own.
q
Ohh, ohk let me try once
I did exactly what is there in docs and link you shared. But I'm getting this error
Copy code
Traceback (most recent call last):
  File "main.py", line 20, in <module>
    cred = Secret("AWS_CREDENTIALS").get()
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 139, in get
    ) from None
ValueError: Local Secret "AWS_CREDENTIALS" was not found.
j
How did you configure the secret?
q
I created the file 
config.toml
added
Copy code
export PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS='{"ACCESS_KEY": "abcdef", "SECRET_ACCESS_KEY": "ghijklmn"}'
j
Ah, yeah. So you have two options for configuring prefect: • Setting keys using environment variables starting with 
PREFECT__
. There are many ways to set environment variables in linux, 
.bashrc
is one. • Writing 
toml
in the 
config.toml
file. In the above you've mixed the two. To set your config var in the toml file you'd write:
Copy code
[context.secrets.AWS_CREDENTIALS]
ACCESS_KEY = "1234"
SECRET_ACCESS_KEY = "5678"
q
I changed my .toml to exactly what you suggested but I'm getting the same error.
Copy code
Traceback (most recent call last):
  File "main.py", line 20, in <module>
    cred = Secret("AWS_CREDENTIALS").get()
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 139, in get
    ) from None
ValueError: Local Secret "AWS_CREDENTIALS" was not found.
j
Where did you put your config file?
The following works fine for me:
Copy code
from prefect import Flow, task
from prefect.tasks.secrets import PrefectSecret

@task
def print_secret(s):
    print(s)


with Flow("test") as flow:
    s = PrefectSecret("AWS_CREDENTIALS")
    print_secret(s)


flow.run()
q
Ok let me try
j
with 
~/.prefect/config.toml
of:
Copy code
[context.secrets.AWS_CREDENTIALS]
ACCESS_KEY = "1234"
SECRET_ACCESS_KEY = "5678"
q
Let me tell you the issue, I was using 
Secret
instead of 
TaskSecret
.
But now I'm facing another issue where it's.... TypeError: run() got an unexpected keyword argument 'credentials'
in S3Upload
j
What version of prefect are you using? That keyword argument does exist.
q
Oh, I was using 0.10.7
so there they've ``aws_credentials_secret``
j
Yeah, providing as a runtime parameter is new in 0.11.0
if possible I recommend upgrading.
q
Yeah, I did it already.
Thank you so much @Jim Crist-Harif. You're Awesome, would like to connect you on LinkedIn: (https://www.linkedin.com/in/tanmay-srivastava-1a9279129/)
j
Glad to help, hope prefect is useful for you. And I don't have a linkedin :).
q
Ohk No issues... Thanks man 🙂
Hey @Jim Crist-Harif can you tell me how to set custom secret?
j
Any field in the 
context.secrets
config section is a valid secret. So the same steps as above, but replace 
AWS_CREDENTIALS
with whatever key you want.
q
Copy code
[context.secrets.CONSTANTS]
BUCKET_NAME = "abcd"
Like if I put it like this and access using
Copy code
a =  Secret("CONSTANTS").get()
print(a)
It should work?
j
Sorry, the 
AWS_CREDENTIALS
secret above contained a dict. If you only want to map a key to a value you'd set:
Copy code
[context.secrets]
KEY1 = "value1"
KEY2 = "value2"
to make 
PrefectSecret("KEY1")
and 
PrefectSecret("KEY2")
work.
q
Ohhh ohkkk
run
method of 
PrefectSecret
is giving error
j
You'll need to post the error for me to help debug.
q
Copy code
Traceback (most recent call last):
  File "main.py", line 23, in <module>
    a =  PrefectSecret("KEY1").run()
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/utilities/tasks.py", line 279, in method
    return run_method(self, *args, **kwargs)
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/tasks/secrets/base.py", line 65, in run
    return _Secret(name).get()
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 143, in get
    ) from None
ValueError: Local Secret "KEY1" was not found.
j
The following works for me:
Copy code
from prefect import Flow, task
from prefect.tasks.secrets import PrefectSecret

@task
def print_secret(s):
    print(s)


with Flow("test") as flow:
    s = PrefectSecret("EXAMPLE")
    print_secret(s)


flow.run()
with the following config at 
~/.prefect/config.toml
Copy code
[context.secrets]
EXAMPLE = "hello"
The error you're seeing indicates that the secret key 
KEY1
wasn't found in the prefect configuration, so you've set the secret incorrectly either in the toml file or as an environment variable.
q
Copy code
Traceback (most recent call last):
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/engine/task_runner.py", line 943, in get_task_run_state
    self.task.run, timeout=self.task.timeout, **raw_inputs
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/utilities/executors.py", line 182, in timeout_handler
    return fn(*args, **kwargs)
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/utilities/tasks.py", line 279, in method
    return run_method(self, *args, **kwargs)
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/tasks/secrets/base.py", line 65, in run
    return _Secret(name).get()
  File "/home/tanmay/.local/lib/python3.6/site-packages/prefect/client/secrets.py", line 143, in get
    ) from None
ValueError: Local Secret "EXAMPLE" was not found.
j
And your config looks like the one I posted and is stored in the proper location at 
~/.prefect/config.toml
?
q
It was a issue with the format, I had AWS_CREDENTIALS secret also in the same file, so formatting got incorrect
j
glad you figured it out
q
Thanks man you're a savior.
4 Views
Open in Slack
PreviousNext
Powered by