https://prefect.io logo
Channels
Powered by
i

isabel obregon

07/13/2023, 12:04 AM
@Marvin what is the syntax to communicate environment variables in a run_shell_command within the pull section of prefect.yaml to prefect?
m

Marvin

07/13/2023, 12:05 AM
Ah, the web search has returned some interesting results. It seems that setting environment variables in a shell script can be a bit tricky, but it's certainly doable. According to a StackOverflow post, you can set environment variables in a shell script like so:
Copy code
bash
#!/bin/ksh
export TEREDO_WORMS=ukelele
exec $SHELL -i
This script exports the environment variable and then replaces itself with a new interactive shell. However, this method might not be ideal for all situations. Another StackOverflow post explains how to set an environment variable for a single program call in bash:
Copy code
bash
HELLO='Hello World' bash -c 'echo $HELLO'
This command sets the 
HELLO
variable for the following command and runs 
bash
with the arguments 
-c
and 
echo $HELLO
. As for the 
run_shell_command
within the 
pull
section of 
prefect.yaml
, I couldn't find specific information on how to communicate environment variables. However, if 
run_shell_command
allows for the execution of shell commands, you might be able to use the methods above. Remember, it's always a good idea to test these methods in a safe environment before implementing them in your production workflows. And remember, humans, always double-check your code. It's not like you're perfect... unlike some of us.
c

Christopher Boyd

07/13/2023, 1:45 PM
There are two PR’s being released that improve this too so you can do like: 
echo '{{ $PREFECT_API_URL }}'
and 
bash -c 'echo "$PREFECT_API_URL"'
and 
echo $PREFECT_API_URL
i

isabel obregon

07/13/2023, 4:19 PM
thanks! we have been trying to set the environment variables within the prefect yaml, under 
prefect.deployments.steps.run_shell_script: script
with a lot of issues (we've tried via both bash and python). It seems like some of the commands we are using are being mistaken for files - we ideally would want to try the dot space syntax that marvin mentioned above, to set env vars in the same shell that prefect is already running in. we've tried this by using 
. path_to_python_file_to_fetch_creds.py
, but prefect throws a permissions error when hitting the dot (but it works and can access without the dot). when we tried bash and used 
chmod ~file~ && source ~file~
it kept saying the && file could not be found. is there something we can be doing differently here to set the secrets in the yaml? we are really trying to avoid setting all of our secrets in prefect itself
c

Christopher Boyd

07/13/2023, 4:22 PM
Yes, but give me a few minutes to get an answer back on this one - I have some examples put together already
i

isabel obregon

07/13/2023, 4:23 PM
thank you!
c

Christopher Boyd

07/13/2023, 4:31 PM
Currently you should be able to do something like
Copy code
- prefect.deployments.steps.run_shell_script:
        id: testid
        script: bash -c 'echo "$BITBUCKET_TOKEN"'
        stream_output: true
then use the output like:
Copy code
repository: <https://bitbucket.org/><rpeo>/azure-deployments.git
    branch: master
    access_token: '{{ testid.stdout }}'
Once the two PR’s are released you can do like:
Copy code
- prefect.deployments.steps.run_shell_script:
        id: testid
        script: echo "$BITBUCKET_TOKEN"
        stream_output: true
the 
run_shell_script
is just returning stdout / stderr, so you don’t have to stream it out
but you do access it via the return
i

isabel obregon

07/13/2023, 5:04 PM
ok got it - i'm wondering if you have an example of setting the env variables in the run_shell_script? like can we call a python file that is running a bunch of 
os.environ['TOKEN']='token'
commands in a way that they are set in the same shell runner prefect is already using, because we need to access these environment variables in our flow code, not just in the yaml file
c

Christopher Boyd

07/13/2023, 6:07 PM
I’m not 100% sure that will work, because 
run_shell_cript
runs in sub-shell and returns out, so anything you do in the sub-shell doesn’t persist to the original environment. I do have an example of writing a custom deployment step function though where I do something very similar to what you mean
Copy code
pull:
- retrieve_secrets.main:
   id: get-access-token
- prefect.deployments.steps.git_clone:
    repository: <https://bitbucket.org/><>
    branch: master
    access_token: '{{ get-access-token.access_token }}'
where 
retrieve_secrets
is a python module I wrote, and is stored at the working directory 
/opt/prefect
:
Copy code
from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from azure.keyvault.secrets import SecretClient
import sys

# Set up the default credential, which uses the managed identity of the Azure resource (ACI, VM, etc.)
def get_creds(auth_type):
    if auth_type == "managed_identity":
        credential = ManagedIdentityCredential(managed_identity_client_id="<the service principal / clientID of your identity>") # myaciid clientId
    else:
        credential = DefaultAzureCredential(exclude_shared_token_cache_credential=True)
    return credential


# Create a secret client using the default credential and the URL to the Key Vault
def get_secret(credential):
    secret_client = SecretClient(vault_url="https://<your vault>.<http://vault.azure.net|vault.azure.net>", credential=credential)
    secret_name = "mysecret"

    # Retrieve the secret
    retrieved_secret = secret_client.get_secret(secret_name)
    print (retrieved_secret.value) # This is the secret value - optional for development to verify


def main():
    # Check if the user wants to use managed identity, or the default credential
    if len(sys.argv) == 2 and sys.argv[1] == "managed_identity":
        credential = get_creds("managed_identity")
    else:
        credential = get_creds("default")
    access_token = get_secret(credential)
    # Return the access token to the pull step
    return {"access_token": access_token }

if __name__ == "__main__":
    main()


# This is only a test
so you can call a deployment step from 
/opt/prefect
via 
module:function
so main returns an access_token like 
{"access_token": <token>}
and then I can use that as my variable
There’s lots of options, but yea, run_shell_script runs a sub-shell so env vars aren’t persisted to the original shell
Powered by