Hi all How do you handle logging if you want to be able to r Prefect Community #prefect-community

Hi all. How do you handle logging if you want to b...

Thomas Hoeck

09/03/2020, 1:14 PM

Hi all. How do you handle logging if you want to be able to run a task separatly for e.g. unit-test:

from prefect import task

import prefect

@task

def extract():

logger = prefect.context.get("logger")

logger.debug("Executing")

return [1,2,3]

extract.run()

as this yields

Copy code

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-25-9ca06496a242> in <module>
     11 
     12 
---> 13 extract.run()

<ipython-input-25-9ca06496a242> in extract()
      7 def extract():
      8     logger = prefect.context.get("logger")
----> 9     logger.debug("Executing")
     10     return [1,2,3]
     11 

AttributeError: 'NoneType' object has no attribute 'debug'

Dylan

09/03/2020, 2:10 PM

Hi @Thomas Hoeck! In the context of a unit test, I would suggest mocking

prefect.context

and the

logger

using

MagicMock

or the like

Dylan

09/03/2020, 2:10 PM

https://docs.python.org/3/library/unittest.mock.html#unittest.mock.MagicMock

Thomas Hoeck

09/03/2020, 2:52 PM

What if just want to run it in a Jupyter Notebook doing my development work? Is there a way to establish a context?

Thomas Hoeck

09/03/2020, 2:53 PM

@Dylan as in I import the task and run it in a Jupyter Notebook with .run()

Dylan

09/03/2020, 2:59 PM

You might be able to

import prefect

and call the logger directly inside your task

Dylan

09/03/2020, 2:59 PM

Let me double check

Dylan

09/03/2020, 3:04 PM

That doesn’t work

Dylan

09/03/2020, 3:05 PM

The only other thing I can think of that would work without changing task code so that it could run outside of a prefect context would be to implement some sort of

Logger

to replace the one provided by the prefect context

Dylan

09/03/2020, 3:05 PM

and then setting the task up like:

Copy code

if prefect.context:
  logger = prefect.context.get("logger")
else:
  logger = LocalLogger()

Dylan

09/03/2020, 3:06 PM

But you’re right, everything else inside a task’s

run

function should work outside of a prefect context

Dylan

09/03/2020, 5:12 PM

Hey Thomas, after a discussion with the team I opened an issue for this: https://github.com/PrefectHQ/prefect/issues/3256

Dylan

09/03/2020, 5:12 PM

Follow this issue for more updates! =]

Iain Dillingham

09/04/2020, 9:13 AM

Thanks for the discussion on this. I encountered the same issue but resolved it by importing

get_logger

from

prefect.utilities.logging

. I get the logger by calling that function rather than through the context object.

Iain Dillingham

09/04/2020, 9:15 AM

get_logger

is also typed, so mypy doesn't complain.

Thomas Hoeck

09/04/2020, 9:31 AM

Nice @Iain Dillingham! But do you get the root logger then or in the context of the task? Still it is mentioned as the suggested way to get logger is https://docs.prefect.io/core/concepts/logging.html#logging-configuration

Iain Dillingham

09/04/2020, 9:34 AM

Ah, good point: it returns the root logger by default. And yes, this isn't the documented way of getting the logger.

👍 1

Thomas Hoeck

09/04/2020, 9:39 AM

@Dylan will follow along 👀

💯 1

Open in Slack

Previous Next