https://prefect.io logo
Title
m

Manuel Ledesma

02/03/2022, 11:14 PM
Hi, how do we setup prefect to use localstack running locally on minikube?
k

Kevin Kho

02/04/2022, 12:37 AM
Hi @Manuel Ledesma, have you seen the instructions on the helm chart README ?
m

Manuel Ledesma

02/04/2022, 11:49 AM
Thank Kevin, I did but there's not an environment variable to replace AWS endpoint URL, It's possible to replace the credentials using environment variables, but there's not a provided variable by AWS, which means that needs to be provided by prefect
I can pushing flows to s3 but the issues is prefect downloading it from localstack s3, we will need to override the client endpoint.
The boto3 cient initialization should be similar to the code below:
if LOCALSTACK is not None:
    CONFIG = botocore.config.Config(retries={'max_attempts': 0})
    s3_client = boto3.client(
        's3',
        region_name='us-east-1',
        endpoint_url=LOCALSTACK,
        config=CONFIG
    )
else:
    s3_client = boto3.client("s3")
that way it will allow local development
I found the way, when initializing the S3 storage, just need to pass the pass the endpoint_url as client_options
client_opts={"endpoint_url": '<http://local-localstack:4566>'}
flow.storage = S3(
        bucket=bucket,
        stored_as_script=True,
        key=key,
        client_options=client_opts
    )
It works