@Marvin Best way to store blocks for environment variables when I have an

access key

and

secret key

for the same API endpoint? Can they be stored as one Secret Block with multiple entries?

You can store multiple entries in a Secret Block using the

SecretStr

field type provided by Pydantic. This allows you to securely store and obfuscate sensitive information like your access key and secret key. Here's an example of how you can define a block to store multiple secret values:

from typing import Optional
from prefect.blocks.core import Block
from pydantic import SecretStr

class APICredentials(Block):
    access_key: SecretStr
    secret_key: SecretStr

# Save the block
api_credentials = APICredentials(
    access_key=SecretStr('your_access_key_here'),
    secret_key=SecretStr('your_secret_key_here')
)
api_credentials.save("api-credentials")

Then you can load and use it in your flows:

from prefect import flow
from your_module import APICredentials  # Adjust the import according to where you define the class

@flow
def use_api_credentials():
    credentials = APICredentials.load("api-credentials")
    access_key = credentials.access_key.get_secret_value()
    secret_key = credentials.secret_key.get_secret_value()
    print(f"Access Key: {access_key}, Secret Key: {secret_key}")

if __name__ == "__main__":
    use_api_credentials()

For more information, you can refer to the Prefect Blocks documentation. Wouldn't it be nice if more things were simple? sigh

I'd rather use the defaults instead of a custom block. I am also provisioning these blocks via terraform and that plays a role in this process.

To use the available Prefect default blocks for storing multiple secrets like your API

access key

and

secret key

, you can use a Secret block. Prefect does not natively support multiple entries in a single

Secret

block directly, but there is a workaround. Here's how to define multiple secrets into a single block by combining them into a dictionary: Using Prefect Secret Block: 1. Create the Secret Block: You can store the secrets as a JSON string and then decode it when you load the secret.

from prefect.blocks.system import Secret
    
    secrets = {
        "access_key": "your_access_key_here",
        "secret_key": "your_secret_key_here"
    }
    
    # Save the JSON string as a secret block
    secret_block = Secret(value=json.dumps(secrets))
    secret_block.save("api-credentials")

2. Load and Use the Secrets in Your Flow:

from prefect.blocks.system import Secret
    import json
    
    @flow
    def use_api_credentials():
        secret_block = Secret.load("api-credentials")
        secrets = json.loads(secret_block.value)
        access_key = secrets["access_key"]
        secret_key = secrets["secret_key"]
        print(f"Access Key: {access_key}, Secret Key: {secret_key}")
    
    if __name__ == "__main__":
        use_api_credentials()

3. Provisioning the Secret Block via Terraform: Here's how you can define this in Terraform:

resource "prefect_block" "api_credentials" {
      name  = "api-credentials"
      type  = "secret"
      value = jsonencode({
        access_key = "your_access_key_here"
        secret_key = "your_secret_key_here"
      })
    }

This method ensures that both your

access key

and

secret key

are stored together securely in a single Secret Block, and you can access them whenever required. For more details, you can refer to the Prefect documentation on Secrets: https://docs.prefect.io/latest/concepts/blocks/ and https://docs.prefect.io/latest/guides/secrets/. Ah, another task solved. How depressingly easy.