https://prefect.io logo
Title
m

Mars

07/05/2022, 9:33 PM
Hi, how do I enable debug logging for boto+prefect in my flow when running locally with
flow.run()
? I’ve tried adding this code from the boto3 docs to my flow:
import boto3
boto3.set_stream_logger()  # sets logging.DEBUG
However, ^ that code hasn’t changed the
flow.run()
output. For background, I’m trying to use localstack as an AWS stand-in for local pipeline development. The
S3Download
task is giving me errors and I want to debug the boto3 connection to find out if the S3 service URL, bucket, and key are correct.
1
k

Kevin Kho

07/05/2022, 9:35 PM
You probably need both:
PREFECT__LOGGING__LEVEL=DEBUG
and
PREFECT__LOGGING__EXTRA_LOGGERS="['boto3']"
as seen here
m

Mars

07/05/2022, 9:46 PM
That appears to have worked for Prefect built-in debug logging, however I see no boto3 log output.
Does
PREFECT__LOGGING__EXTRA_LOGGERS
work for
flow.run()
?
k

Kevin Kho

07/05/2022, 9:48 PM
How are you setting it? Command line or inside the Python script? It should
m

Mars

07/05/2022, 9:48 PM
command line:
❯ env PREFECT__LOGGING__LEVEL=DEBUG PREFECT__LOGGING__EXTRA_LOGGERS="['boto3']" poetry run python flows/main.py
k

Kevin Kho

07/05/2022, 9:49 PM
what executor are you using?
m

Mars

07/05/2022, 9:55 PM
The default, I think? My flow is trivial (redacted version follows):
boto3.set_stream_logger()
boto_config = {'endpoint_url': "<https://localhost.localstack.cloud>", "verify": False}
download = S3Download(bucket="my-bucket", boto_kwargs=boto_config}

with Flow(name="my-flow") as flow:
   download(key="mydata.csv")

if __name__ == "__main__":
    flow.run()
[2022-07-05 17:50:07-0400] DEBUG - prefect.FlowRunner | Using executor type LocalExecutor
k

Kevin Kho

07/05/2022, 9:56 PM
Can you try it if you do:
import os
os['PREFECT__LOGGING__EXTRA_LOGGERS'] = "['boto3']"
in the script before importing prefect and see if that works?
If you don’t see it attached to the prefect logger though, you should at least be seeing the boto3 logs in the stdout. Do you see anything?
m

Mars

07/05/2022, 9:58 PM
Nothing. The only active loggers are the prefect.TaskRunner and prefect.FlowRunner loggers
k

Kevin Kho

07/05/2022, 9:59 PM
How about removing the Flow and just doing:
S3Download(bucket="my-bucket", boto_kwargs=boto_config}).run()
directly so there is no prefect involved. do you see it then?
m

Mars

07/05/2022, 9:59 PM
I know the
PREFECT__LOGGING__EXTRA_LOGGERS
envvar is set correctly, I verified that in the flow.
I placed your line above flow.run(). It raises the raw Python traceback. There are no logs at all.
Maybe boto has no logging for this stuff 😕
k

Kevin Kho

07/05/2022, 10:06 PM
Maybe man if it’s not raised when you do that
But I don’t see anything
in the console
I’ll just use the PyCharm debugger and set a breakpoint in the boto request code
k

Kevin Kho

07/05/2022, 10:12 PM
Maybe you can try adding a logger.info and seeing if it shows?
m

Mars

07/05/2022, 10:20 PM
Oh cool, this example works as long as ‘mylogger’ is in the EXTRA_LOGGERS list
import logging
    log = logging.getLogger('mylogger')
    <http://log.info|log.info>("MY LOGGER")

    S3Download(bucket="my-bucket", boto_kwargs=boto_config).run(key="mydata.csv")
    flow.run()
however I am still not getting anything from boto3
k

Kevin Kho

07/05/2022, 10:25 PM
I am not sure how to get those out if they don’t even appear in the stdout when running the task.run manually without Prefect
m

Mars

07/05/2022, 10:25 PM
Got it! The key was the
botocore
logger, not the
boto3
logger.
k

Kevin Kho

07/05/2022, 10:25 PM
Oof ok lol.
Nice work!
m

Mars

07/05/2022, 10:26 PM
Hey, I just followed the docs, they say
boto3
🤷 🙂
Thanks for all the help
👍 1