Hello! I was trying to migrate my legacy data pipeline into a prefect flow, by refactoring the `main...

Goh Rui Zhi

07/21/2021, 8:57 AM

Hello! I was trying to migrate my legacy data pipeline into a prefect flow, by refactoring the

main()

of each job into a task. This runs fine, but the problem is that the logs for the functions outside of each the

main()

method will not be logged using

prefect.context

and hence not appear on the prefect UI. Is there a way for Prefect to capture all logs without explicitly changing the logger for each of the sub functions to use that of

prefect.context

? I know that making each job into a subprocess and then redirecting the stdout into Prefect's logger will work, but would like to avoid using subprocesses in general. Besides, this method also seems very hackish haha.

Goh Rui Zhi

07/21/2021, 9:15 AM

The each task calls subfunctions from other files like shown below:

Copy code

@task
def process_data():
    ...
    while start_date < end_date:
         processor.run()
    ...

The logs for

run()

will not be captured by prefect.

Amanda Wee

07/21/2021, 9:51 AM

You may be able to attach extra loggers: https://docs.prefect.io/core/concepts/logging.html#extra-loggers

Goh Rui Zhi

07/21/2021, 12:37 PM

Thanks for the reply! Throughout my projects, in every file, there is a logger initialised this way:

Copy code

import logging
logger = logging.getLogger(__name__)

Can you advise me on how I can get those logs to display on prefect server?

Kevin Kho

07/21/2021, 2:17 PM

Hey @Goh Rui Zhi, my understanding is that doing it that way preserves the hierarchy of the logging in module. So if you have logs that are like

module.submodule

and

module.submodule2

. Attaching

module

to the extra loggers like in the link Amanda showed will work in adding all.

Goh Rui Zhi

07/22/2021, 3:04 AM

I see, I will try that in a bit. Thank you both :)

Goh Rui Zhi

07/22/2021, 5:30 AM

Hmmm somehow my local prefect configs are not sent over to my the prefect-server when I do flow.register(). I tried directly setting the environment variables on the prefect-server UI to include my extra loggers and it worked perfectly. What am I missing here?

Kevin Kho

07/22/2021, 2:44 PM

Could you show me how you set the environment variables? Maybe the syntax was just a bit off?

Goh Rui Zhi

07/23/2021, 10:01 AM

I simply did

Copy code

export PREFECT__LOGGING__EXTRA_LOGGERS="['my_module']"

The prefect server is running on GKE.

Kevin Kho

07/23/2021, 1:21 PM

The environment variables from your local machine won’t get carried over. It needs to be set where the Flow is running. Could you try doing it through the RunConfig with

KubernetesRun(env={"PREFECT__LOGGING__EXTRA_LOGGERS": "['my_module']"})

👍 1

Goh Rui Zhi

07/24/2021, 2:32 AM

That's exactly what I needed! Thanks again @Kevin Kho :)

👍 1

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.