q

    Qin XIA

    1 year ago
    Hello, I do need some information for passing "endpoint_url" to prefect.tasks.aws.s3.S3List.run to change default endpoint URL. Is it possible in PREFECT? Like `aws s3 ls --endpoint-url https://s3.xxx.xxx.xxxx/`in aws cli. Thx a lot
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Qin XIA, it doesn’t look like it’s supported. I was checking, do you know if it’s available in boto3?
    Bryan Cusatis

    Bryan Cusatis

    1 year ago
    Did you try passing endpoint_url to boto_kwargs?
    That would be my first guess at least
    q

    Qin XIA

    1 year ago
    @Kevin Kho Yes it's available in boto3 session. That's what i use now.
    Bryan Cusatis

    Bryan Cusatis

    1 year ago
    It doesn’t look like S3 List has the kwarg override like the other ones. Not sure if this is possible.
    Kevin Kho

    Kevin Kho

    1 year ago
    Yeah I think you would have to make your own task similar to S3 list here
    q

    Qin XIA

    1 year ago
    @Bryan Cusatis I just read code,
    s3_client = get_boto_client("s3", credentials=credentials)
    , only credentials is passed and in
    get_boto_client
    it only gets "ACCESS_KEY", "SECRET_ACCESS_KEY", "SESSION_TOKEN". So I think i would have to use boto3 at this moment. Thanks a lot
    @Kevin Kho yes Thx for your response.
    Bryan Cusatis

    Bryan Cusatis

    1 year ago
    Yeah the download one has overrides. Not sure why list doesn’t.
    Could be a good PR. 😉
    Kevin Kho

    Kevin Kho

    1 year ago
    I agree. We’d accept a PR for this 🙂