Mars
07/05/2022, 9:33 PMflow.run()
?
I’ve tried adding this code from the boto3 docs to my flow:
import boto3
boto3.set_stream_logger() # sets logging.DEBUG
However, ^ that code hasn’t changed the flow.run()
output.
For background, I’m trying to use localstack as an AWS stand-in for local pipeline development. The S3Download
task is giving me errors and I want to debug the boto3 connection to find out if the S3 service URL, bucket, and key are correct.Kevin Kho
07/05/2022, 9:35 PMPREFECT__LOGGING__LEVEL=DEBUG
and
PREFECT__LOGGING__EXTRA_LOGGERS="['boto3']"
as seen hereMars
07/05/2022, 9:46 PMPREFECT__LOGGING__EXTRA_LOGGERS
work for flow.run()
?Kevin Kho
07/05/2022, 9:48 PMMars
07/05/2022, 9:48 PM❯ env PREFECT__LOGGING__LEVEL=DEBUG PREFECT__LOGGING__EXTRA_LOGGERS="['boto3']" poetry run python flows/main.py
Kevin Kho
07/05/2022, 9:49 PMMars
07/05/2022, 9:55 PMboto3.set_stream_logger()
boto_config = {'endpoint_url': "<https://localhost.localstack.cloud>", "verify": False}
download = S3Download(bucket="my-bucket", boto_kwargs=boto_config}
with Flow(name="my-flow") as flow:
download(key="mydata.csv")
if __name__ == "__main__":
flow.run()
[2022-07-05 17:50:07-0400] DEBUG - prefect.FlowRunner | Using executor type LocalExecutor
Kevin Kho
07/05/2022, 9:56 PMimport os
os['PREFECT__LOGGING__EXTRA_LOGGERS'] = "['boto3']"
in the script before importing prefect and see if that works?Mars
07/05/2022, 9:58 PMKevin Kho
07/05/2022, 9:59 PMS3Download(bucket="my-bucket", boto_kwargs=boto_config}).run()
directly so there is no prefect involved. do you see it then?Mars
07/05/2022, 9:59 PMPREFECT__LOGGING__EXTRA_LOGGERS
envvar is set correctly, I verified that in the flow.Kevin Kho
07/05/2022, 10:06 PMMars
07/05/2022, 10:10 PMKevin Kho
07/05/2022, 10:12 PMMars
07/05/2022, 10:20 PMimport logging
log = logging.getLogger('mylogger')
<http://log.info|log.info>("MY LOGGER")
S3Download(bucket="my-bucket", boto_kwargs=boto_config).run(key="mydata.csv")
flow.run()
Kevin Kho
07/05/2022, 10:25 PMMars
07/05/2022, 10:25 PMbotocore
logger, not the boto3
logger.Kevin Kho
07/05/2022, 10:25 PMMars
07/05/2022, 10:26 PMboto3
🤷 🙂