Should I need to pass the credentials as a param to the function read_csv? Or are they read automatically from the Cloud?
Currently I'm getting this error:
"botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden"
Thanks!
a
Anna Geller
05/30/2022, 2:34 PM
Try creating a boto3 session in which you use those credentials
from pyarrow.fs import S3FileSystem
# or instead of pyarrow.fs
import s3fs
import pyarrow.dataset as ds
import duckdb
con = duckdb.connect()
fs = S3FileSystem(access_key="my_access_key", secret_key="my_secret_key", region="eu-1/frankfurt")
# or
fs = s3fs.S3FileSystem(anon=False, key="my_access_key", secret="my_secret_key")
history = ds.dataset("findata/forex_pros/history/D", partitioning=["exchange"], filesystem=fs)
aapl = con.execute("SELECT * FROM history WHERE symbol_id=6408").df()
m
Mateo Merlo
06/06/2022, 12:24 PM
Thanks @Volker L !!
🙂 1
v
Volker L
06/06/2022, 12:30 PM
You are welcome. Hope this short introduction is helpful. Contact me, if you need some more input.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.