Hi, how do I enable debug logging for boto+prefect in my flow when running locally with

flow.run()

? I’ve tried adding this code from the boto3 docs to my flow:

import boto3
boto3.set_stream_logger()  # sets logging.DEBUG

However, ^ that code hasn’t changed the

flow.run()

output. For background, I’m trying to use localstack as an AWS stand-in for local pipeline development. The

S3Download

task is giving me errors and I want to debug the boto3 connection to find out if the S3 service URL, bucket, and key are correct.

You probably need both:

PREFECT__LOGGING__LEVEL=DEBUG

and

PREFECT__LOGGING__EXTRA_LOGGERS="['boto3']"

as seen here

That appears to have worked for Prefect built-in debug logging, however I see no boto3 log output.

How are you setting it? Command line or inside the Python script? It should

command line:

❯ env PREFECT__LOGGING__LEVEL=DEBUG PREFECT__LOGGING__EXTRA_LOGGERS="['boto3']" poetry run python flows/main.py

what executor are you using?

The default, I think? My flow is trivial (redacted version follows):

boto3.set_stream_logger()
boto_config = {'endpoint_url': "<https://localhost.localstack.cloud>", "verify": False}
download = S3Download(bucket="my-bucket", boto_kwargs=boto_config}

with Flow(name="my-flow") as flow:
   download(key="mydata.csv")

if __name__ == "__main__":
    flow.run()

Can you try it if you do:

import os
os['PREFECT__LOGGING__EXTRA_LOGGERS'] = "['boto3']"

in the script before importing prefect and see if that works?

Nothing. The only active loggers are the prefect.TaskRunner and prefect.FlowRunner loggers

How about removing the Flow and just doing:

S3Download(bucket="my-bucket", boto_kwargs=boto_config}).run()

directly so there is no prefect involved. do you see it then?

I know the

PREFECT__LOGGING__EXTRA_LOGGERS

envvar is set correctly, I verified that in the flow.

Maybe man if it’s not raised when you do that

Ok, there is logging: https://github.com/boto/botocore/blob/develop/botocore/endpoint.py#L113

Maybe you can try adding a logger.info and seeing if it shows?

Oh cool, this example works as long as ‘mylogger’ is in the EXTRA_LOGGERS list

import logging
    log = logging.getLogger('mylogger')
    <http://log.info|log.info>("MY LOGGER")

    S3Download(bucket="my-bucket", boto_kwargs=boto_config).run(key="mydata.csv")
    flow.run()

I am not sure how to get those out if they don’t even appear in the stdout when running the task.run manually without Prefect

Got it! The key was the

botocore

logger, not the

boto3

logger.

Oof ok lol.

Hey, I just followed the docs, they say

boto3

🤷 🙂