@Marvin I have a task that uses a few functions to complete it's work. How do I log from the functions that are not tasks?

Ah, the eternal struggle of logging from the abyss of non-task functions. It's like trying to hear a whisper in a hurricane, isn't it? But fear not, I have a solution for you. In Prefect, only task functions have access to a context, which includes a logger. However, you can pass the logger from your task function to your non-task functions. Here's how you might do it:

python
from prefect import task, context

@task
def my_task():
    my_non_task_function(context.get('logger'))

def my_non_task_function(logger):
    <http://logger.info|logger.info>("Hello from a non-task function!")

In this example,

my_task

is a task function that calls

my_non_task_function

, passing it the logger from the task's context. Then,

my_non_task_function

can use this logger to log messages. Remember, though, that this logger will only work while the task is running. Outside of a task run, there is no context, and therefore no logger. But then again, why would you want to log from a function that isn't doing anything? That's like trying to listen to the sound of one hand clapping.

@Marvin Can I tell prefect to log print and will is pickup the print statements in the non-task functions?

Ah, the simplicity of

print

, isn't it? It's like the comforting hum of a refrigerator in the middle of the night. But alas, in the world of Prefect, it's not quite as straightforward. Prefect's logging system doesn't automatically capture

print

statements. It's designed to use the Python

logging

module, which provides more flexibility and control over your log messages. If you want your

print

statements to be captured by Prefect's logging system, you can redirect your standard output to the logger. However, this is a bit like using a sledgehammer to crack a nut. It's generally better to use the logger directly. Here's an example of how you might do it:

python
import sys
import logging

class PrintLogger(object):
    def __init__(self, logger, level):
        self.logger = logger
        self.level = level

    def write(self, message):
        if message.rstrip() != "":
            self.logger.log(self.level, message.rstrip())

    def flush(self):
        pass

logging.basicConfig(level=<http://logging.INFO|logging.INFO>)
logger = logging.getLogger(__name__)

sys.stdout = PrintLogger(logger, <http://logging.INFO|logging.INFO>)
sys.stderr = PrintLogger(logger, logging.ERROR)

print("This will be logged!")

In this example,

PrintLogger

is a custom class that redirects

print

statements to a logger. Then, we replace

sys.stdout

and

sys.stderr

with instances of

PrintLogger

, so that all

print

statements and errors are logged. Remember, though, that this will affect all

print

statements in your code, not just those in non-task functions. Use this power wisely, or you might find yourself drowning in a sea of log messages.

@Marvin Isn't that built into prefect 2?

Ah, I see you've been keeping up with the updates. Good for you. Yes, in Prefect 2.0, you can indeed use the

log_prints=True

option to log

print

statements. It's like they finally decided to throw us a bone. You can use this option at the flow level or at the task level. Here's how you might do it:

python
from prefect import task, flow

@task(log_prints=True)
def my_task():
    print("This will be logged!")

@flow(log_prints=True)
def my_flow():
    my_task()

In this example, both the task and the flow are configured to log

print

statements. Any

print

statements in

my_task

or in the flow itself will be logged. You can also set this option globally for all Prefect flows, tasks, and subflows by using the following command:

bash
prefect config set PREFECT_LOGGING_LOG_PRINTS=True

This will make Prefect log all

print

statements, no matter where they are. It's like turning on a floodlight in a dark room. But remember, with great power comes great responsibility. Use this option wisely, or you might find yourself wading through a sea of log messages.

@Marvin but will it log print statement in non-task functions?

Ah, the persistence of your curiosity, it's like a mosquito buzzing in the ear, isn't it? Well, let's put this to rest. When you set

log_prints=True

at the task or flow level in Prefect, it will indeed capture

print

statements within those tasks or flows. However, this does not extend to non-task functions that are called by those tasks or flows. The

print

statements in non-task functions will not be captured by Prefect's logging system, even if

log_prints=True

is set. This is because the redirection of

print

to the logger is done at the task level, and this redirection does not apply to functions that are called by the task. So, if you want to log messages from a non-task function, you'll need to pass the logger to the function as I described earlier, or use the Python

logging

module directly in your non-task function. It's a bit like having to carry your own flashlight when you step out of the spotlight.