https://prefect.io
Join Slack
Hello everyone, I´m trying to use gitlab storage in one test-flow…I stored the gitlab token inside c...
# prefect-server
r
Hello everyone, I´m trying to use gitlab storage in one test-flow…I stored the gitlab token inside config.toml under 
[context.secrets]
The flow seems to get the token as the error that raises the flow run in the UI says that the value was not found -> 
Failed to load and execute Flow's environment: ValueError('Local Secret "RZFXuBfesc_XXXXXXXXX" was not found.')
Prefect and Gitlab are two ec2 instances and the sg of gitlab one allow ssh and https from prefect’ s ip.
Copy code
import prefect
from prefect import task, Flow, Parameter
from prefect.executors import LocalExecutor
from prefect.run_configs import LocalRun
from prefect.storage import GitLab
import requests, os

from prefect.client.secrets import Secret
s = Secret("PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN")

@task()
def overon_get_missing(days):
    url = "<https://XXXXXXXXX.execute-api.eu-west-1.amazonaws.com/prod/api/v1/getMissing/>"+ str(days)
    response = <http://requests.post|requests.post>(url)
    return response

with Flow("overon_get_missing",run_config=LocalRun(),executor=LocalExecutor()) as flow:
    days = Parameter('days')
    result = overon_get_missing(days)
    print(result)

flow.storage = GitLab(
    repo="dmed/soa/prefect/overon_get_missing_test",                            # name of repo
    path="overon_get_missing_gitlab.py",                    # location of flow file in repo
    access_token_secret=s.get(),   # name of personal access token secret
    host="<https://gitlab.test.net>",
    ref = "master"
)

overon_get_missing.run(days=30)
n
Hi @Raúl Mansilla - where is the 
config.toml
file located from which you're trying to pull the secret?
r
Hello @nicholas, 
./prefect/config.toml
I mean, home directory of the main user…then the path I wrote…
n
It looks like you're passing in the secret value at registration time instead of the secret reference with `s.get()`; I believe gitlab storage is looking only for the reference 
GITLAB_ACCESS_TOKEN
r
I also tried doing an export and trying to get as GITLAB_ACCESS_TOKEN, but the same happens….what do you suggest?
c
Another option that I am using is AWS Secrets Manager. Its handy/cheap if you don't want to store the secrets on the server.
Copy code
import boto3 

security_client = boto3.client('secretsmanager')
security_responese = security_client.get_secret_value(SecretId='name_of_my_secret_in_aws')
security_dict = json.loads(security_responese['SecretString'])

print(security_dict['username'])
👍 1
r
@codingtacos that is one thing I wanted to do 🙂 thx!
c
Nice! I use it on all my flows 🙂
🙌 1
n
Try this @Raúl Mansilla :
Copy code
flow.storage = GitLab(
    # ...
    access_token_secret='GITLAB_ACCESS_TOKEN',
    # ...
)
✅ 1
r
mmm, O thought I tested it before, now it seems to be working better though now I have a timeout error trying to conect via https…I will check de sg rules. Thanks @nicholas
👍 1
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by