https://prefect.io logo
Channels
Powered by
r

Raed

06/09/2021, 7:15 AM
Hello, what would be the best way to pass a bitbucket access token to prefect-sever? (I.e. Set a secret in prefect-server) I setup the server using the instructions in the most recent

stream▾

When trying the following for a given flow
Copy code
flow.run_configs = KubernetesRun(
    image=<image>,
    env={
        "PREFECT__CONTEXT__SECRETS__BITBUCKET_ACCESS_TOKEN": os.environ[
            "BITBUCKET_ACCESS_TOKEN"
        ]
    },
    image_pull_policy="Always",
)
flow.storage = Bitbucket(
    project=<project>,
    repo=<repo>,
    path="flows/example_flow.py",
    access_token_secret="BITBUCKET_ACCESS_TOKEN",
)
I get the following error on the UI
Copy code
Failed to load and execute Flow's environment: ValueError('Local Secret "BITBUCKET_ACCESS_TOKEN" was not found.')
Wouldn't the access token secret have been set in the run config? I also have the same environment variable in a custom docker image being used by the run config
k

Kevin Kho

06/09/2021, 2:48 PM
Hey @Raed, Just confirming that registration works but just execution fails right?
So the environment variables passed from the RunConfig are applied after the Flow is loaded from storage. In this specific case, you’ll need to set it on the agent with 
prefect agent ____ start --env BITBUCKET_ACCESS_TOKEN=token
for the agent to be able to load the Flow from storage.
👍 1
r

Raed

06/09/2021, 4:43 PM
Correct, registration worked but it fails on execution
Thanks I will try that, I also tried to set it in inside my values.yaml within the Perfect config but that did not work. To confirm, 'prefect agent kubernetes start' would automatically apply to my k8s cluster if I've entered the Apollo IP into my local config?
k

Kevin Kho

06/09/2021, 4:48 PM
Yes the agent will hit the Apollo IP and be responsible for starting the k8s jobs.
r

Raed

06/09/2021, 7:10 PM
So that would be equivalent to changing the values.yaml as follows?
Copy code
agent:
  # enabled determines if the Prefect Kubernetes agent is deployed
  enabled: true # So I dont have to manually start the agent and pass the variable 

  # prefectLabels defines what scheduling labels (not K8s labels) should
  # be associated with the agent
  prefectLabels: []

  # image configures the container image for the agent deployment
  image:
    repository: <Private Image> # imaage contains BITBUCKET_ACCESS_TOKEN as an environemnt variable 
    tag: latest
    pullPolicy: Always
    pullSecrets: []
So if the image is being pulled after the flow is loaded, I could just use a universal executor given my values.yaml contains all the needed params?
Copy code
flow.run_configs = KubernetesRun(
    image="<Private Image>:latest",
)
Also, if
Copy code
agent:
   enabled: true
does that mean running
Copy code
prefect agent ____ start --env BITBUCKET_ACCESS_TOKEN=token
will be ignored? Because kubernetes already has an agent service?
k

Kevin Kho

06/09/2021, 8:08 PM
Hey @Raed, I talked to some team members about this. Can you try the original code but use 
flow.run_config
instead of 
flow.run_configs
?
r

Raed

06/09/2021, 8:46 PM
Hey @Kevin Kho, good catch, I switched it to run_config but still get the same error, I figure this must be because of what you stated above regarding RunConfig being applied afterwards?
Copy code
[9 June 2021 4:44pm]: Failed to load and execute Flow's environment: ValueError('Local Secret "BITBUCKET_ACCESS_TOKEN" was not found.')
Any word on if the above values.yaml approach is alright? It'd be ideal to just have everything managed from within the helm chart, and would the above make the KubernetesRun object redundant?
k

Kevin Kho

06/09/2021, 8:56 PM
So when I chatted with the engineers, the conclusion I got (which is different from I told you earlier) is that the RunConfig env variables will be enough to pull down the Flow from Storage. If you want to do it from Kubernetes, that contains that environment variable 
PREFECT__CONTEXT__SECRETS__BITBUCKET_ACCESS_TOKEN
, then yes you don’t need to start a new one with the command line
Sorry, you’ll want 
BITBUCKET_ACCESS_TOKEN
in the environment instead of the other one.
r

Raed

06/09/2021, 9:03 PM
So I created the env variable 
BITBUCKET_ACCESS_TOKEN
created an interactive shell onto the kubernetes agent, verified the key was correct with inside the pod
Copy code
echo $BITBUCKET_ACCESS_TOKEN
but still get the same error
Copy code
[9 June 2021 5:01pm]: Failed to load and execute Flow's environment: ValueError('Local Secret "BITBUCKET_ACCESS_TOKEN" was not found.')
So you're saying I should change the name of the env variable from 
BITBUCKET_ACCESS_TOKEN
to 
PREFECT__CONTEXT__SECRETS__BITBUCKET_ACCESS_TOKEN
? In both
Copy code
flow.run_config = KubernetesRun(
    image=<private docker image>,
    env={"PREFECT__CONTEXT__SECRETS__BITBUCKET_ACCESS_TOKEN": os.environ["KS_BITBUCKET_ACCESS_TOKEN"]},
)
& the docker image?
k

Kevin Kho

06/09/2021, 9:07 PM
Just on the values.yaml should be good. We shouldn’t need the env anymore in KubernetesRun. Yes what you tried is what I was suggesting.
Checking more things
Do you only have one agent on? Just wanna make sure the right agent is picking up the Flow
r

Raed

06/09/2021, 9:09 PM
Correct, only one agent
k

Kevin Kho

06/09/2021, 9:17 PM
Can you try removing the access token secret from the Bitbucket storage like:
Copy code
flow.run_config = KubernetesRun(
    image=<image>,
    env={
        "BITBUCKET_ACCESS_TOKEN": os.environ[
            "BITBUCKET_ACCESS_TOKEN"
        ]
    },
    image_pull_policy="Always",
)
flow.storage = Bitbucket(
    project=<project>,
    repo=<repo>,
    path="flows/example_flow.py",
)
r

Raed

06/09/2021, 9:23 PM
Tried that just now, authorization failed
Copy code
[9 June 2021 5:22pm]: Failed to load and execute Flow's environment: HTTPError('401 Client Error: Unauthorized for url:<URL>')
Also building an image with 
PREFECT__CONTEXT__SECRETS__BITBUCKET_ACCESS_TOKEN
just in case, read somewhere that it expects all secrets in that format and would make sense why it's failing to find env
k

Kevin Kho

06/09/2021, 9:29 PM
Are you trying to hit Bitbucket Cloud or Server?
r

Raed

06/09/2021, 9:35 PM
Bitbucket Cloud
Will report back later on if putting the below in the image worked 
PREFECT__CONTEXT__SECRETS__BITBUCKET_ACCESS_TOKEN
k

Kevin Kho

06/09/2021, 9:38 PM
It looks like the env variable was picked up but the token is unauthorized on Bitbucket.
I’m looking at the documentation and it looks like the ACCESS_TOKEN_SECRET was for Bitbucket Server only. Cloud uses username and password for authentication. Looking at the code if you can use the token with Cloud
My bad. I should have asked if Bitbucket Cloud or Server earlier. Cloud requires username and password to authenticate here
You need 
BITBUCKET_CLOUD_USERNAME
and 
BITBUCKET_CLOUD_APP_PASSWORD
as the environment variables for this
r

Raed

06/10/2021, 5:06 PM
Copy code
It looks like the env variable was picked up but the token is unauthorized on Bitbucket.
The token isn't picked up at all, have used this token in the pass successfully and I believe the error message above says it doesn't exist.
Thanks for catching that, I will try adding those as variables in my image and reporting back
k

Kevin Kho

06/10/2021, 5:10 PM
The error is happening because the code directs 
BITBUCKET_ACCESS_TOKENS
to Bitbucket Server, which causes the “Unauthorized” error, but you’re using Bitbucket Cloud. The token probably works for cloud but the way Bitbucket Storage is coded directs Access Tokens to server and username + password to Cloud.
👍 1
Do you have a workspace for Cloud you can pass to the Storage?
Am reading the code again and it uses Prefect Cloud if you have a workspace, but uses Prefect Server if you don’t have one and I didnt see a workspace in the codesnippet earlier.
r

Raed

06/10/2021, 5:26 PM
Yes I have a workspace, I've updated the code as follows
Copy code
flow.run_config = KubernetesRun(
    image=<private_image>,
    env={
        "BITBUCKET_CLOUD_USERNAME": os.environ["BITBUCKET_CLOUD_USERNAME"],
        "BITBUCKET_CLOUD_APP_PASSWORD": os.environ["BITBUCKET_ACCESS_TOKEN"],
    },
   


# * bitbucket storage
flow.storage = Bitbucket(
    project=<project>,
    repo=<repo>,
    workspace=<workspace>,
    cloud_username_secret="BITBUCKET_CLOUD_USERNAME",
    cloud_app_password_secret="BITBUCKET_CLOUD_APP_PASSWORD",
    path="flows/flow.py",
)
Gets error
Copy code
[10 June 2021 1:26pm]: Failed to load and execute Flow's environment: ValueError('Local Secret "BITBUCKET_CLOUD_USERNAME" was not found.')
Do you think that might have to do with formatting of environment variables so they can be used as secrets? That is
Copy code
flow.run_config = KubernetesRun(
    image=<private_image>,
    env={
        "PREFECT__CONTEXT__SECRETS__BITBUCKET_CLOUD_USERNAME": os.environ["BITBUCKET_CLOUD_USERNAME"],
        "PREFECT__CONTEXT__SECRETS__BITBUCKET_CLOUD_APP_PASSWORD": os.environ["BITBUCKET_ACCESS_TOKEN"],
    },
I think this may have worked, am getting a new error when using the new naming convention, assuming it must not be able to find the flow in the repo, is there an easy way to see the full stacktrace on the UI?
Copy code
[10 June 2021 1:32pm]: Failed to load and execute Flow's environment: FileNotFoundError(2, 'No such file or directory')
k

Kevin Kho

06/10/2021, 5:36 PM
Was gonna say I think the second one will work with, bPREFECT__CONTEXT__SECRETS__ but was verifying. Do you have DEBUG level logs enabled? I think you’ll get more logs if you do that. This specific error means than it can’t find the flow at the place you pointed in Bitbucket. Did you supply a path?
The path here is where it will look for the flow to download.
Copy code
flow = Flow("my-flow")
flow.storage = Bitbucket(
    project="my.project", repo="my.repo", path="/flows/flow.py", ref="my-branch"
)
What version of Prefect are you on also?
r

Raed

06/10/2021, 5:38 PM
Copy code
prefect -- 0.14.21
k

Kevin Kho

06/10/2021, 5:41 PM
0.14.19 had issues with CLI registration but you should be fine. We just need to provide the path to the file in Bitbucket
r

Raed

06/10/2021, 5:57 PM
Seems like it tries in downloading the flow, but as soon as it begins it throws this error
Copy code
Failed to load and execute Flow's environment: KeyError('BITBUCKET_CLOUD_USERNAME')
So now the actual environment variable is missingm but that's odd because my flow originally had this line upon registration
Copy code
flow.run_config = KubernetesRun(
    image=<image>,
    env={
        "PREFECT__CONTEXT__SECRETS__BITBUCKET_CLOUD_USERNAME": os.environ[
            "BITBUCKET_CLOUD_USERNAME"
        ],
        "PREFECT__CONTEXT__SECRETS__BITBUCKET_CLOUD_APP_PASSWORD": os.environ[
            "BITBUCKET_ACCESS_TOKEN"
        ],
        "PREFECT__LOGGING__LEVEL": "DEBUG",
    },
    image_pull_policy="IfNotPresent",
)
When the flow runs I presume this must cause that error by trying to run os.environ['BITBUCKET_CLOUD_USERNAME'], but passing debug doesn't provide the full stack trace, any idea how I can get it? Also shouldn't the above line automatically adjust the agent service on kubernetes to have those environment variables? Or do I have to restart the pod manually ?
k

Kevin Kho

06/10/2021, 5:59 PM
It actually says 
KS_BITBUCKET_CLOUD_USERNAME
? That’s so weird. Let me chat with the team about full traceback.
r

Raed

06/10/2021, 6:03 PM
sorry should say "BITBUCKET_CLOUD_USERNAME"
k

Kevin Kho

06/10/2021, 6:04 PM
Can I see how you set up the RunConfig? Are you doing it inside the 
with Flow() as flow:
block or outside?
r

Raed

06/10/2021, 6:07 PM
Outside, after the context manager
k

Kevin Kho

06/10/2021, 6:08 PM
Ok that should be good. Will ask the team about debugging.
I’ll try to get a working example for Bitbucket today I think I have some time
r

Raed

06/10/2021, 6:41 PM
that would be awesome, thanks again for all your help! I suspect the issue is what you mentioned in the beginning is whats causing this error
Copy code
So the environment variables passed from the RunConfig are applied after the Flow is loaded from storage.
So the flow is loaded, the run config is executed, but the env variables are not there so it fails, trying again with a rebuilt image that has the correct env vars
k

Kevin Kho

06/10/2021, 6:44 PM
I was wrong when I said that. I have a working script with Bitbucket. I will DM you because I have a password written out.
3 Views
Powered by