Sarhan
08/07/2024, 5:42 AMMarvin
08/07/2024, 5:43 AMstorage_options
argument as strings in a dictionary:
1. Create the AWS Credentials Block:
First, create an AWS credentials block. Here's an example using Python code:
python
import os
from prefect_aws import AwsCredentials
my_aws_creds = AwsCredentials(
aws_access_key_id="your_access_key_id",
aws_secret_access_key=os.environ.get("MY_AWS_SECRET_ACCESS_KEY"),
)
my_aws_creds.save(name="my-aws-creds-block", overwrite=True)
2. Load the Credentials Block in Your Code:
Next, load the AWS credentials block and pass them as storage options to `df.to_parquet()`:
python
from prefect import flow
from prefect_aws import AwsCredentials
@flow
def my_flow():
# Load AWS credentials from the block
aws_creds = AwsCredentials.load("my-aws-creds-block")
# Prepare storage options dictionary
storage_options = {
"key": aws_creds.aws_access_key_id,
"secret": aws_creds.aws_secret_access_key,
}
# Your DataFrame code here
df = ... # Assuming you have a DataFrame instance
# Write to Parquet with storage_options
df.to_parquet("<s3://your-bucket/path>", storage_options=storage_options)
if __name__ == "__main__":
my_flow()
Please replace "your_access_key_id"
, os.environ.get("MY_AWS_SECRET_ACCESS_KEY")
, and "<s3://your-bucket/path>"
with your specific values.
Links for more details:
- Guide to creating and using blocks: https://docs.prefect.io/latest/guides/moving-data/
- Example for loading and using blocks: https://raw.githubusercontent.com/zzstoatzz/prefect-code-examples/main/examples/tasks/submitting.py
If you need further details or run into issues, feel free to ask!