Thread
#prefect-community
    Bruno Murino

    Bruno Murino

    1 year ago
    Hi everyone — I’m trying to use the prefect S3List task but I’m struggling with the AWS authentication, since it appears the S3List task doesn’t accept boto_kwargs nor a boto_session, and I need to specify different profiles for my task. Has anyone faced this problem before?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Bruno Murino, I have not seen people use this Task before. Most of our Task Library is community contributed and is left at a very general level so it may not always fit you needs. In this case, I would just use boto3 directly and make a task like:
    @task
    def get_images() -> List[str]:
        s3 = boto3.resource('s3', region_name='us-east-2')
        bucket = s3.Bucket('label-studio-raw-images')
        images = []
        for bucket_object in bucket.objects.all():
            images.append(bucket_object.key)
        return images
    Bruno Murino

    Bruno Murino

    1 year ago
    yea that’s what I ended up doing, thanks!
    Kevin Kho

    Kevin Kho

    1 year ago
    If you feel the task library can be better, contributions are most welcome 🙂
    Bruno Murino

    Bruno Murino

    1 year ago
    yea I’ll definitely try and contribute! 🙂