@Marvin How do I access my aws credentials stored in a block? I would like to pass them to df.to_parquet()’s storage_options arg as strings in a dict

Here's how you can access your AWS credentials stored in a block and pass them to `df.to_parquet()`'s

storage_options

argument as strings in a dictionary: 1. Create the AWS Credentials Block: First, create an AWS credentials block. Here's an example using Python code:

python
import os
from prefect_aws import AwsCredentials

my_aws_creds = AwsCredentials(
    aws_access_key_id="your_access_key_id",
    aws_secret_access_key=os.environ.get("MY_AWS_SECRET_ACCESS_KEY"),
)

my_aws_creds.save(name="my-aws-creds-block", overwrite=True)

2. Load the Credentials Block in Your Code: Next, load the AWS credentials block and pass them as storage options to `df.to_parquet()`:

python
from prefect import flow
from prefect_aws import AwsCredentials

@flow
def my_flow():
    # Load AWS credentials from the block
    aws_creds = AwsCredentials.load("my-aws-creds-block")

    # Prepare storage options dictionary
    storage_options = {
        "key": aws_creds.aws_access_key_id,
        "secret": aws_creds.aws_secret_access_key,
    }
    
    # Your DataFrame code here
    df = ... # Assuming you have a DataFrame instance

    # Write to Parquet with storage_options
    df.to_parquet("<s3://your-bucket/path>", storage_options=storage_options)

if __name__ == "__main__":
    my_flow()

Please replace

"your_access_key_id"

,

os.environ.get("MY_AWS_SECRET_ACCESS_KEY")

, and

"<s3://your-bucket/path>"

with your specific values. Links for more details: - Guide to creating and using blocks: https://docs.prefect.io/latest/guides/moving-data/ - Example for loading and using blocks: https://raw.githubusercontent.com/zzstoatzz/prefect-code-examples/main/examples/tasks/submitting.py If you need further details or run into issues, feel free to ask!