https://prefect.io
Join Slack
Is it possible to change the logging-level of parameter tasks? For flows that have many parameters (...
# ask-community
j
Is it possible to change the logging-level of parameter tasks? For flows that have many parameters (and most with default values), it'd be nice to remove them from the info-level logs, where they can be overwhelming.
k
Hey @Jack Sundberg, I don’t see a way without setting the global setting. Also looking at the source , there’s not explicit info level logging statement, which means that it’s inheriting all of those so I think this would be hard to change unfortunately.
j
Okay, bummer but that's fine haha. Thanks for checking though!
Also I just did a quick check myself -- the logger calls are actually in the task_runner class. So I'd have to open a pull request for this and just add a few if-statements
k
I honestly am not sure we’d accept PR’s into the 
task_runner
specifically because it’s very core to Prefect. I do have an idea actually, you can explicitly create a filter for the Python logger to hide some logs. Have you seen this before?
Copy code
import prefect
from prefect import task, Flow
import logging

# Creating a filter
class SecureFilter(logging.Filter):
    def filter(self, rec):
        if 'tes' in rec.msg:
            return 0
        return 1

@task
def abc():
    logger = prefect.context.get("logger")
    logger.addFilter(SecureFilter())
    <http://logger.info|logger.info>("teting")        # gets shows
    <http://logger.info|logger.info>("testing")       # gets hidden
    return 1

@task
def bcd():
    logger = prefect.context.get("logger")
    logger.addFilter(SecureFilter())
    <http://logger.info|logger.info>("teting")        # gets shows
    <http://logger.info|logger.info>("testing")       # gets hidden
    return 1

with Flow("test") as flow:
    abc()
    bcd()

flow.run()
j
oh woah, thank you!
Am I able to do these two lines outside of a task? Or the context only accessible inside a task?
Copy code
logger = prefect.context.get("logger")
logger.addFilter(SecureFilter())
k
For you though it’s a bit trickier because you need it in the Flow block:
Copy code
with Flow("test") as flow:
    logger = prefect.context.get("logger")
    logger.addFilter(SecureFilter())
    abc()
    bcd()
so that it affect Parameters, but then the Flow block is evaluated during build time so I think your logger will only be configured when you register. In order for it to apply to the flow run, you need to 
store_as_script=True
in your storage.
I believe you should have context if stored as script. How are you storing this Flow?
j
I use 
ModuleStorage
, which I believe behaves like 
store_as_script=True
k
That sounds right. Let me give this a test also.
Ok it didnt work for me. Let me ask the team if there’s another way to do this
j
Yeah, I'm struggling to get it working too. If there's no easy way, don't worry about it. It probably won't be worth the effort lol
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by