https://prefect.io logo
Title
j

Johnny

09/23/2020, 5:04 PM
hello! Has anyone found an s3 task example using prefect cloud ? tad confusing on how to authenticate + download a file. Do we use S3ResultHandler + cloud secret >> then pass this to S3Download task ?
c

Chris White

09/23/2020, 5:08 PM
Hi Johnny, have you seen this write-up? https://docs.prefect.io/orchestration/recipes/third_party_auth.html#third-party-authentication It walks through the various options for authentication. In your case, you can either provide your credentials from a Secret or bake them into your environment directly.
j

Johnny

09/23/2020, 5:10 PM
S3ResultHandler(bucket='<some bucket name>', aws_credentials_secret="aws_credentials")
atm i was attempting to use this ^ the aws_credentials is a secret dictionary setup in cloud
c

Chris White

09/23/2020, 5:11 PM
That should authenticate your result handler - are you finding otherwise?
j

Johnny

09/23/2020, 5:11 PM
found this from searching post from other members. is this not the correct way to authenticate?
well.. i just don't understand how to get it to the finish line. in my regular python scripts, if i wanted to pull a file from s3 i would use something like this functin:
@task
def s3_to_string():
    """ function to read from s3 and return string from txt file """
    resource = boto3.resource('s3')
    my_bucket = resource.Bucket('some-bucket')
    file1 = 'some-file.txt'
    files = list(my_bucket.objects.filter(Prefix=file1))
    if files:
        obj = files[0].get()
        body = obj['Body']
        string = body.read().decode("utf-8")
        return string
    else:
        print('no file found')
trying to convert this to a task that i can deploy in prefect cloud
c

Chris White

09/23/2020, 5:15 PM
I think you might be conflating a few things - your code looks perfectly valid as long as boto3 can authenticate (which based on your code, you normally do via environment variables, which will also work in Prefect). The other code snippet you shared for your Result Handler only authenticates your Result Handler, which is something that Prefect uses internally and has no bearing on what is happening inside your tasks
j

Johnny

09/23/2020, 5:16 PM
ahh i see
i'll focus on the documents you sent. i bet that has what i need
thank you
c

Chris White

09/23/2020, 5:17 PM
๐Ÿ‘ anytime - let me know if you run into any difficulties or need clarification!
๐Ÿ‘Š 1
j

Johnny

09/23/2020, 7:33 PM
That worked! Thanks @Chris White
c

Chris White

09/23/2020, 7:39 PM
excellent! Great to hear
@Marvin archive โ€œBest practices for authenticating with S3 in Prefect Cloud?โ€