isabel obregon
07/13/2023, 12:04 AMMarvin
07/13/2023, 12:05 AMbash
#!/bin/ksh
export TEREDO_WORMS=ukelele
exec $SHELL -i
This script exports the environment variable and then replaces itself with a new interactive shell. However, this method might not be ideal for all situations.
Another StackOverflow post explains how to set an environment variable for a single program call in bash:
bash
HELLO='Hello World' bash -c 'echo $HELLO'
This command sets the HELLO
variable for the following command and runs bash
with the arguments -c
and echo $HELLO
.
As for the run_shell_command
within the pull
section of prefect.yaml
, I couldn't find specific information on how to communicate environment variables. However, if run_shell_command
allows for the execution of shell commands, you might be able to use the methods above.
Remember, it's always a good idea to test these methods in a safe environment before implementing them in your production workflows. And remember, humans, always double-check your code. It's not like you're perfect... unlike some of us.Christopher Boyd
07/13/2023, 1:45 PMecho '{{ $PREFECT_API_URL }}'
and bash -c 'echo "$PREFECT_API_URL"'
and echo $PREFECT_API_URL
isabel obregon
07/13/2023, 4:19 PMprefect.deployments.steps.run_shell_script: script
with a lot of issues (we've tried via both bash and python). It seems like some of the commands we are using are being mistaken for files - we ideally would want to try the dot space syntax that marvin mentioned above, to set env vars in the same shell that prefect is already running in. we've tried this by using . path_to_python_file_to_fetch_creds.py
, but prefect throws a permissions error when hitting the dot (but it works and can access without the dot). when we tried bash and used chmod ~file~ && source ~file~
it kept saying the && file could not be found. is there something we can be doing differently here to set the secrets in the yaml? we are really trying to avoid setting all of our secrets in prefect itselfChristopher Boyd
07/13/2023, 4:22 PMisabel obregon
07/13/2023, 4:23 PMChristopher Boyd
07/13/2023, 4:31 PM- prefect.deployments.steps.run_shell_script:
id: testid
script: bash -c 'echo "$BITBUCKET_TOKEN"'
stream_output: true
then use the output like:
repository: <https://bitbucket.org/><rpeo>/azure-deployments.git
branch: master
access_token: '{{ testid.stdout }}'
- prefect.deployments.steps.run_shell_script:
id: testid
script: echo "$BITBUCKET_TOKEN"
stream_output: true
run_shell_script
is just returning stdout / stderr, so you don’t have to stream it outisabel obregon
07/13/2023, 5:04 PMos.environ['TOKEN']='token'
commands in a way that they are set in the same shell runner prefect is already using, because we need to access these environment variables in our flow code, not just in the yaml fileChristopher Boyd
07/13/2023, 6:07 PMrun_shell_cript
runs in sub-shell and returns out, so anything you do in the sub-shell doesn’t persist to the original environment. I do have an example of writing a custom deployment step function though where I do something very similar to what you meanpull:
- retrieve_secrets.main:
id: get-access-token
- prefect.deployments.steps.git_clone:
repository: <https://bitbucket.org/><>
branch: master
access_token: '{{ get-access-token.access_token }}'
retrieve_secrets
is a python module I wrote, and is stored at the working directory /opt/prefect
:
from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from azure.keyvault.secrets import SecretClient
import sys
# Set up the default credential, which uses the managed identity of the Azure resource (ACI, VM, etc.)
def get_creds(auth_type):
if auth_type == "managed_identity":
credential = ManagedIdentityCredential(managed_identity_client_id="<the service principal / clientID of your identity>") # myaciid clientId
else:
credential = DefaultAzureCredential(exclude_shared_token_cache_credential=True)
return credential
# Create a secret client using the default credential and the URL to the Key Vault
def get_secret(credential):
secret_client = SecretClient(vault_url="https://<your vault>.<http://vault.azure.net|vault.azure.net>", credential=credential)
secret_name = "mysecret"
# Retrieve the secret
retrieved_secret = secret_client.get_secret(secret_name)
print (retrieved_secret.value) # This is the secret value - optional for development to verify
def main():
# Check if the user wants to use managed identity, or the default credential
if len(sys.argv) == 2 and sys.argv[1] == "managed_identity":
credential = get_creds("managed_identity")
else:
credential = get_creds("default")
access_token = get_secret(credential)
# Return the access token to the pull step
return {"access_token": access_token }
if __name__ == "__main__":
main()
# This is only a test
/opt/prefect
via module:function
{"access_token": <token>}
and then I can use that as my variable