<@ULVA73B9P> When I want an S3 bucket as cache sto...
# ask-marvin
m
@Marvin When I want an S3 bucket as cache storage for my flow runs, do I need to provide an aws secret key? what if i want prefect to use the role of the ec2 instance it is running on?
m
thought for 266.1 seconds
Short answer: No — you don’t need to provide an AWS secret key. Prefect’s AWS integrations use boto3 under the hood, so they follow the standard AWS credential provider chain. If your flow runs on an EC2 instance with an attached IAM role (instance profile) that has S3 permissions, Prefect will use that role automatically as long as you don’t explicitly pass credentials. What to set up - Ensure the instance role has the necessary S3 permissions (e.g., s3:PutObject, s3:GetObject, s3:ListBucket for your bucket/prefix). - If you’re running in a container on EC2/ECS/EKS, make sure the runtime can access its role credentials (IMDSv2 on EC2, task role on ECS, IRSA on EKS). boto3 will pick these up automatically. Two easy ways to use S3 for results/caching in Prefect 3.x 1) Using RemoteFileSystem (no explicit AWS creds)
Copy code
from prefect import flow
from prefect.filesystems import RemoteFileSystem

s3_store = RemoteFileSystem(basepath="<s3://my-bucket/prefect/results/>")

@flow(result_storage=s3_store, persist_result=True)
def my_flow():
    return "hello"
2) Using the Prefect AWS S3Bucket block (also works with instance role)
Copy code
from prefect import flow
from prefect_aws.s3 import S3Bucket

# Do not pass AwsCredentials — boto3 will use the instance role
s3_store = S3Bucket(bucket_name="my-bucket", basepath="prefect/results/")

@flow(result_storage=s3_store, persist_result=True)
def my_flow():
    return "hello"
Notes - Prefect writes persisted task/flow results and caches to the configured result storage. If you enable caching (via
cache_policy
or cache settings on tasks/flows), those artifacts will be saved to S3 using the same
result_storage
. - Only pass explicit
AwsCredentials
if you want to override the default credential chain (e.g., for local dev). On EC2 with a proper instance role, omit them. Docs for reference: - Prefect AWS integrations: prefect-aws docs - Blocks and configuration: Blocks in Prefect 3 - Results and result storage: prefect.results API If you share where you’re running (EC2 directly, ECS, or EKS), I can tailor the IAM policy and any runtime-specific setup details.