Hi all, I have having an issue with our logging implementation changes are not holding when registered to and running in Prefect Cloud. Essentially, the custom log handler (which is just opentelemetry-sdk) enforces a new

_log_record_context_injector

function to modify the names of

PREFECT_LOG_RECORD_ATTRIBUTES

for our purposes which is then added to the root Prefect logger. This function call occurs in a Task class I created which is set as an

upstream_task.

When this is run locally, all log records have the corrected attribute names. When run in Cloud, only the task run to initialize the log handler enforces the changes. Given that all tasks should inherent from the root logger, why are the changes not being implemented across the tasks/flow? Further info: we are using KubernetesRun with Docker flow storage. Also the changes hold when the implementation is run locally.

So the logger is part of

context

and modifications to the

context

only live inside that task. So for example if you do:

@task
def abc():
    prefect.context.x =3
    return ..

This x does not exist in downstream tasks. Instead, the better attempt here would be to attach it outside of the tasks and Flow if possible

logger = prefect.context.get("logger")
logger.addHandler()

@task
def abc():
    ...

and then used script based storage so that it’s not serialized and it’s run during script start. Still unsure if this will work. Were you the one who asked about the

PREFECT_LOG_RECORD_ATTRIBUTES

before as well? I think those are hardcoded

Yes I am the one that asked about

PREFECT_LOG_RECORD_ATTRIBUTES

before. I'm basically aliasing those values which does work. I'm curious though why this wouldn't work to access the root Prefect logger for changes?

# Set up custom logger LogRecord Factory
        logging.setLogRecordFactory(_log_record_context_injector)

        # Attach OTLP handler to root Prefect logger
        logger = get_logger()
        logger.addHandler(handler)
        <http://logger.info|logger.info>("OTEL Log Handler is set up!")

I will need to ask some people to get more detailed answers. I don’t know this much detail about the logger myself. The attempt with

logger = prefect.context.get("logger")
logger.addHandler()

was that inside or outside a task?

Likely within a task if I remember correctly which I learned would only affect that task context. That's why I started using

get_logger

from the utility module instead. I think I should also clarify that the log handler I add to the prefect logger IS attaching. The issue I'm encountering is that the

_log_record_context_injector

function to modify the names of

PREFECT_LOG_RECORD_ATTRIBUTES

is somehow not holding. I find that curious because all tasks should be calling

get_logger(self.name)

which should yield a child logger from root. So it seems to me that the original

_log_record_context_injector

is being applied to those child loggers. Not sure why that is if the root logger has been changed

Looks like we don’t have an existing issue to track OpenTelemetry. Would you be willing to write one in the repo? I can then direct future users to chime in there as well.