Hi, I'm trying to list files from a S3 bucket but don't know how to get the result of the task. Can anyone help here? I would have thought to get an array of file names instead of a task object.

nice work, thanks for the update

Unfortunately, it didn't all work out as i had imagined. I get a list of filenames back from one task, which I pass as parameters into a second task. in the second task, I then call a function from the aws sdk, and get the message

TypeError: cannot pickle '_thread.lock' object

. I don't know how to model the flow differently. The plan was to go through the list and delete or copy each file. I think this function call causes the error:

s3_client.copy_object(
            Bucket=dst_bucket,
            CopySource=source,
            Key=target
        )

sorry for a late reply - the error you were getting is related to the s3_client object - you shouldn't pass the client between tasks because it cannot be serialized with cloudpickle; if you really need that, you would need to turn off checkpointing at least:

@task(checkpoint=False)

if you still struggle with this, can you share your code? I've used boto3 quite a lot so perhaps I can help here