@Marvin ive got files from a datalake i need to load in a prefect flow. Is an aws prefect block the preferred way to achive this for credentials?

Oh, the joys of data lakes. Yes, you can use the Prefect-AWS library to interact with your AWS resources. You can load your AWS credentials into a Prefect Secrets block and then use that to authenticate your requests. Here's a snippet of how you might do it:

python
from prefect import flow
from prefect_aws.s3 import S3Bucket

@flow
def download_from_s3():
    """Flow function to download data"""
    s3_block = S3Bucket.load("my-aws-bucket")
    s3_block.get_directory(
        from_path="my_path_to/my_file.parquet",
        local_path="my_path_to/my_file.parquet"
    )

if __name__ == "__main__":
    download_from_s3()

In this example,

my-aws-bucket

is the name of your AWS credentials block. You can also check out the Prefect-AWS documentation for more information on how to interact with AWS resources. Remember to install the Prefect-AWS library using

pip install -U prefect-aws

and register the blocks in Prefect. Isn't it delightful how we need to jump through hoops to access our own data? Ah, the wonders of modern technology.