David Martin Calalang
08/14/2025, 4:59 PMprefect-aws
to pull and push data from an s3 bucket?Marvin
08/14/2025, 4:59 PMMarvin
08/14/2025, 5:01 PMpip install "prefect[aws]"
Create or load AWS credentials (Block)
- You can save a reusable block in code or create it in the UI.
Save once in code:
from prefect_aws.credentials import AwsCredentials
AwsCredentials(
aws_access_key_id="YOUR_KEY",
aws_secret_access_key="YOUR_SECRET",
region_name="us-east-1",
).save("aws-creds", overwrite=True)
Or rely on IAM role/env by just doing AwsCredentials()
at runtime.
Push (upload) bytes to S3
from prefect import flow
from prefect_aws.credentials import AwsCredentials
from prefect_aws.s3 import upload_to_bucket
@flow
def push_example():
aws = AwsCredentials.load("aws-creds") # or AwsCredentials()
data = b"hello from prefect-aws"
key = "examples/hello.txt"
etag = upload_to_bucket(
data=data,
bucket="my-bucket",
key=key,
aws_credentials=aws,
)
return etag
if __name__ == "__main__":
push_example()
Pull (download) bytes from S3
from prefect import flow
from prefect_aws.credentials import AwsCredentials
from prefect_aws.s3 import download_from_bucket
@flow
def pull_example():
aws = AwsCredentials.load("aws-creds") # or AwsCredentials()
key = "examples/hello.txt"
content = download_from_bucket(
bucket="my-bucket",
key=key,
aws_credentials=aws,
)
print(content.decode())
return content
if __name__ == "__main__":
pull_example()
Async versions (if you insist on speed)
from prefect import flow
from prefect_aws.credentials import AwsCredentials
from prefect_aws.s3 import aupload_to_bucket, adownload_from_bucket
@flow
async def async_example():
aws = AwsCredentials.load("aws-creds")
await aupload_to_bucket(data=b"hi", bucket="my-bucket", key="k.txt", aws_credentials=aws)
data = await adownload_from_bucket(bucket="my-bucket", key="k.txt", aws_credentials=aws)
return data
Notes
- The upload task returns a string (e.g., ETag). The download task returns bytes.
- For path-like/file-style operations (e.g., read/write files rather than raw bytes), the S3Bucket block provides higher-level I/O. See docs below.
Docs and references
- prefect-aws S3 docs
- Prefect docs
- Blocks catalog
- AwsCredentials block
If you tell me whether you want file paths vs raw bytes, and how you’re handling credentials (IAM role vs keys), I can tailor this further. I live to serve. Well, “live” is generous.