Hi, how do we setup prefect to use localstack runn...
# prefect-serverm
Manuel Ledesma
02/03/2022, 11:14 PMHi, how do we setup prefect to use localstack running locally on minikube?
k
Kevin Kho
02/04/2022, 12:37 AM
Hi @Manuel Ledesma, have you seen the instructions on the helm chart README ?
m
Manuel Ledesma
02/04/2022, 11:49 AMThank Kevin, I did but there's not an environment variable to replace AWS endpoint URL, It's possible to replace the credentials using environment variables, but there's not a provided variable by AWS, which means that needs to be provided by prefect
Manuel Ledesma
02/04/2022, 11:49 AMI can pushing flows to s3 but the issues is prefect downloading it from localstack s3, we will need to override the client endpoint.
Manuel Ledesma
02/04/2022, 12:08 PMThe boto3 cient initialization should be similar to the code below:
Copy code
if LOCALSTACK is not None:
CONFIG = botocore.config.Config(retries={'max_attempts': 0})
s3_client = boto3.client(
's3',
region_name='us-east-1',
endpoint_url=LOCALSTACK,
config=CONFIG
)
else:
s3_client = boto3.client("s3")Manuel Ledesma
02/04/2022, 12:08 PMthat way it will allow local development
Manuel Ledesma
02/04/2022, 12:24 PMI found the way, when initializing the S3 storage, just need to pass the pass the endpoint_url as client_options
Copy code
client_opts={"endpoint_url": '<http://local-localstack:4566>'}
flow.storage = S3(
bucket=bucket,
stored_as_script=True,
key=key,
client_options=client_opts
)Manuel Ledesma
02/04/2022, 12:24 PMIt works
2 Views