https://prefect.io
Join Slack
Hi, how do we setup prefect to use localstack running locally on minikube?
# prefect-server
m
Hi, how do we setup prefect to use localstack running locally on minikube?
k
Hi @Manuel Ledesma, have you seen the instructions on the helm chart README ?
m
Thank Kevin, I did but there's not an environment variable to replace AWS endpoint URL, It's possible to replace the credentials using environment variables, but there's not a provided variable by AWS, which means that needs to be provided by prefect
I can pushing flows to s3 but the issues is prefect downloading it from localstack s3, we will need to override the client endpoint.
The boto3 cient initialization should be similar to the code below:
Copy code
if LOCALSTACK is not None:
    CONFIG = botocore.config.Config(retries={'max_attempts': 0})
    s3_client = boto3.client(
        's3',
        region_name='us-east-1',
        endpoint_url=LOCALSTACK,
        config=CONFIG
    )
else:
    s3_client = boto3.client("s3")
that way it will allow local development
I found the way, when initializing the S3 storage, just need to pass the pass the endpoint_url as client_options
Copy code
client_opts={"endpoint_url": '<http://local-localstack:4566>'}
flow.storage = S3(
        bucket=bucket,
        stored_as_script=True,
        key=key,
        client_options=client_opts
    )
It works
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by