g

    Goh Rui Zhi

    1 year ago
    Hello! I was trying to migrate my legacy data pipeline into a prefect flow, by refactoring the
    main()
    of each job into a task. This runs fine, but the problem is that the logs for the functions outside of each the
    main()
    method will not be logged using
    prefect.context
    and hence not appear on the prefect UI. Is there a way for Prefect to capture all logs without explicitly changing the logger for each of the sub functions to use that of
    prefect.context
    ? I know that making each job into a subprocess and then redirecting the stdout into Prefect's logger will work, but would like to avoid using subprocesses in general. Besides, this method also seems very hackish haha.
    The each task calls subfunctions from other files like shown below:
    @task
    def process_data():
        ...
        while start_date < end_date:
             processor.run()
        ...
    The logs for
    run()
    will not be captured by prefect.
    Amanda Wee

    Amanda Wee

    1 year ago
    g

    Goh Rui Zhi

    1 year ago
    Thanks for the reply! Throughout my projects, in every file, there is a logger initialised this way:
    import logging
    logger = logging.getLogger(__name__)
    Can you advise me on how I can get those logs to display on prefect server?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Goh Rui Zhi, my understanding is that doing it that way preserves the hierarchy of the logging in module. So if you have logs that are like
    module.submodule
    and
    module.submodule2
    . Attaching
    module
    to the extra loggers like in the link Amanda showed will work in adding all.
    g

    Goh Rui Zhi

    1 year ago
    I see, I will try that in a bit. Thank you both πŸ˜ƒ
    Hmmm somehow my local prefect configs are not sent over to my the prefect-server when I do flow.register(). I tried directly setting the environment variables on the prefect-server UI to include my extra loggers and it worked perfectly. What am I missing here?
    Kevin Kho

    Kevin Kho

    1 year ago
    Could you show me how you set the environment variables? Maybe the syntax was just a bit off?
    g

    Goh Rui Zhi

    1 year ago
    I simply did
    export PREFECT__LOGGING__EXTRA_LOGGERS="['my_module']"
    The prefect server is running on GKE.
    Kevin Kho

    Kevin Kho

    1 year ago
    The environment variables from your local machine won’t get carried over. It needs to be set where the Flow is running. Could you try doing it through the RunConfig with
    KubernetesRun(env={"PREFECT__LOGGING__EXTRA_LOGGERS": "['my_module']"})
    g

    Goh Rui Zhi

    1 year ago
    That's exactly what I needed! Thanks again @Kevin Kho πŸ˜ƒ