hello! Has anyone found an s3 task example using p...
# prefect-community
j
hello! Has anyone found an s3 task example using prefect cloud ? tad confusing on how to authenticate + download a file. Do we use S3ResultHandler + cloud secret >> then pass this to S3Download task ?
c
Hi Johnny, have you seen this write-up? https://docs.prefect.io/orchestration/recipes/third_party_auth.html#third-party-authentication It walks through the various options for authentication. In your case, you can either provide your credentials from a Secret or bake them into your environment directly.
j
Copy code
S3ResultHandler(bucket='<some bucket name>', aws_credentials_secret="aws_credentials")
atm i was attempting to use this ^ the aws_credentials is a secret dictionary setup in cloud
c
That should authenticate your result handler - are you finding otherwise?
j
found this from searching post from other members. is this not the correct way to authenticate?
well.. i just don't understand how to get it to the finish line. in my regular python scripts, if i wanted to pull a file from s3 i would use something like this functin:
Copy code
@task
def s3_to_string():
    """ function to read from s3 and return string from txt file """
    resource = boto3.resource('s3')
    my_bucket = resource.Bucket('some-bucket')
    file1 = 'some-file.txt'
    files = list(my_bucket.objects.filter(Prefix=file1))
    if files:
        obj = files[0].get()
        body = obj['Body']
        string = body.read().decode("utf-8")
        return string
    else:
        print('no file found')
trying to convert this to a task that i can deploy in prefect cloud
c
I think you might be conflating a few things - your code looks perfectly valid as long as boto3 can authenticate (which based on your code, you normally do via environment variables, which will also work in Prefect). The other code snippet you shared for your Result Handler only authenticates your Result Handler, which is something that Prefect uses internally and has no bearing on what is happening inside your tasks
j
ahh i see
i'll focus on the documents you sent. i bet that has what i need
thank you
c
๐Ÿ‘ anytime - let me know if you run into any difficulties or need clarification!
๐Ÿ‘Š 1
j
That worked! Thanks @Chris White
c
excellent! Great to hear
@Marvin archive โ€œBest practices for authenticating with S3 in Prefect Cloud?โ€