I am trying to return list of object from aws s3 bucket. I am getting this error when using it in prefect flow “TypeError: can’t pickle _thread.lock “objects using s3_client.list_objects(bucket,prefix)
Otherwise its returning correct output without prefect on jupyter notebook
05/16/2022, 1:29 PM
Do you want to move your flow code to Prefect 2.0? This is easier to do in 2.0 because there's no DAG requirement.
your issue is because your Prefect 1.0 flow gets serialized at registration time and you can't serialize the S3 boto3 client as it's not cloud-pickle serializable, so you would need to move the s3 client object from Flow to your task that lists S3 objects
05/16/2022, 1:43 PM
i have done the same still showing this error, but if i use s3_client.list_objects in the flow its returning the metadata but only “Contents:” is missing , and request id host id , content type and all other keys are in the output