https://prefect.io
Join Slack
Hello, I do need some information for passing "endpoint_url" to prefect.tasks.aws.s3.S3List.run to c...
# ask-community
q
Hello, I do need some information for passing "endpoint_url" to prefect.tasks.aws.s3.S3List.run to change default endpoint URL. Is it possible in PREFECT? Like `aws s3 ls --endpoint-url https://s3.xxx.xxx.xxxx/`in aws cli. Thx a lot
✅ 1
k
Hi @Qin XIA, it doesn’t look like it’s supported. I was checking, do you know if it’s available in boto3?
b
Did you try passing endpoint_url to boto_kwargs?
That would be my first guess at least
q
@Kevin Kho Yes it's available in boto3 session. That's what i use now.
b
It doesn’t look like S3 List has the kwarg override like the other ones. Not sure if this is possible.
k
Yeah I think you would have to make your own task similar to S3 list here
q
@Bryan Cusatis I just read code, 
s3_client = get_boto_client("s3", credentials=credentials)
, only credentials is passed and in 
get_boto_client
it only gets "ACCESS_KEY", "SECRET_ACCESS_KEY", "SESSION_TOKEN". So I think i would have to use boto3 at this moment. Thanks a lot
@Kevin Kho yes Thx for your response.
b
Yeah the download one has overrides. Not sure why list doesn’t.
Could be a good PR. ;-)
👍 1
k
I agree. We’d accept a PR for this 🙂
11 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by