i have tasks that call operators/connectors that i’ve defined with their own loggers. but even when i set
log_stdout=True
those logs don’t show. any advice on how to include logs from those other loggers?
k
Kevin Kho
03/08/2022, 7:04 PM
Have you seen the page on extra loggers ? Are you on DaskExecutor?
l
Lana Dann
03/08/2022, 7:09 PM
nah i’m not using a dask executor. right now i have something like this:
Copy code
import logging
import logging.config
logging.config.fileConfig("logging.conf")
logger = logging.getLogger(__name__)
class SnowflakeConnector:
and then in my task i’ll do something like
Copy code
from connectors.snowflake import SnowflakeConnector
@task
def my_task():
s = SnowflakeConnector()
s.some_method()
so that means for every new logger i initiate with a new name, i need to add the name to the
extra_loggers
section of the config?
k
Kevin Kho
03/08/2022, 7:30 PM
I would recommend passing it as a list in the RunConfig as an environment variable honestly if you can do that, but I think the answer to your question is yes
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.