https://prefect.io logo
Channels
Powered by
c

Charles Leung

08/17/2023, 4:17 PM
Hi Team, Is it possible to have each flow run logged to a separate .log file instead of having one prefect.log? Many thanks!!
m

Moe

08/17/2023, 4:32 PM
I want to know this too. Btw, have you figured out how to save the logs locally?
c

Charles Leung

08/17/2023, 6:01 PM
in your logging.yml, you can set it to save all your flow logs inot a prefect.log locally. but its not separated which is what ive been trying to figure out 😕
m

Moe

08/17/2023, 7:46 PM
How do you do that? I added a file handler according to this article and specified my logging handlers to use this handler but it still isn't saving it locally for me.
Copy code
file:
level: 0
class: logging.handlers.TimedRotatingFileHandler
filename: /var/log/prefect.log
when: 'D'
interval: 1
backupCount: 7
formatter: standard
Can you perhaps share with me your logging.yaml file?
c

Charles Leung

08/17/2023, 8:06 PM
looks similar to yours. check ur file name?
Copy code
handlers:
    file:
        level: 0
        # The handlers we define here will output all logs they receieve by default
        # but we include the `level` so it can be overridden by environment
        class: logging.handlers.RotatingFileHandler
        # Update filename to the location you want to store Prefect's log files. 
        # This location must be writable by the user account that runs Prefect.
        filename: C:\Users\qtask\prefect.log
        # maximum 10MB per log file
        maxBytes: 10485760
        # keep the last 5 log files in addition to the current log file
        backupCount: 5
        formatter: standard
also make sure y our environment variable is pointing to the right logging.yml file
✅ 1
m

Moe

08/17/2023, 8:11 PM
Thanks a lot. 
prefect server
is the handler responsible for creating logs in the UI, right?
Copy code
loggers:
    prefect:
        level: "${PREFECT_LOGGING_LEVEL}"

    prefect.extra:
        level: "${PREFECT_LOGGING_LEVEL}"
        handlers: [prefect server, file]

    prefect.flow_runs:
        level: NOTSET
        handlers: [prefect server, file]

    prefect.task_runs:
        level: NOTSET
        handlers: [prefect server, file]

    prefect.infrastructure:
        level: "${PREFECT_LOGGING_LEVEL}"
        handlers: [prefect server, file]
I have the loggers set this way, but it isn't generating any logs.
Ok I got logs saved locally. Thanks @Charles Leung. Now hoping someone will come along to advise us about separate log files.
🙌 1
c

Charles Leung

08/18/2023, 6:16 PM
@Prefect bump - any help would be appreciated
✅ 1
j

Jake Kaplan

08/18/2023, 7:22 PM
m

Moe

08/18/2023, 7:22 PM
@Jake Kaplan This thread is for saving individual log files per flow run. Is that possible too?
j

Jake Kaplan

08/18/2023, 7:23 PM
ah apologies I misread
Heres one way that works, although I'm sure there is a probably a better way:
Copy code
from prefect import flow, task
from prefect.logging import get_run_logger
from logging import FileHandler
from prefect.runtime import flow_run


def _get_logger():
    handler = FileHandler(filename=f"{flow_run.name}.log")
    logger = get_run_logger()
    # prefect uses a log adapter, so add to the underlying logger
    original_logger = logger.logger
    handler.setFormatter(original_logger.root.handlers[0].formatter)
    original_logger.addHandler(handler)
    return logger

@task
def my_task():
    logger = _get_logger()
    <http://logger.info|logger.info>("Hello from my task!")


@flow
def my_flow():
    logger = _get_logger()
    <http://logger.info|logger.info>("Hello from my Flow!")
    my_task()


my_flow()
🙌 2
m

Moe

08/18/2023, 8:06 PM
Interesting. So with this approach, we don't even need to override the logging.yml file correct?
@Jake Kaplan Amazing, thank you.
j

Jake Kaplan

08/18/2023, 8:10 PM
that is correct! Like I said there is a probably a better way to do it, but i'm not positive we offer the right hook at the moment to do it per flow run, given the point where the logger is configured relative to when a flow run exists
c

Charles Leung

08/22/2023, 11:06 PM
Hey @Jake Kaplan Thanks for helping us out here. I tried running the code snippet you sent above, and I keep getting this error now. Its causing all of my other flows to stop being able to run ad-hoc:
Copy code
Traceback (most recent call last):
  File "//v-tm-qeq-05/PrefectScripts/cleung/Script/logging.py", line 1, in <module>
    from prefect import flow, task, get_run_logger
  File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\prefect\__init__.py", line 37, in <module>
    from prefect.states import State
  File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\prefect\states.py", line 9, in <module>
    import anyio
  File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\anyio\__init__.py", line 103, in <module>
    from ._core._fileio import AsyncFile, Path, open_file, wrap_file
  File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\anyio\_core\_fileio.py", line 24, in <module>
    from .. import to_thread
  File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\anyio\to_thread.py", line 7, in <module>
    from .abc import CapacityLimiter
  File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\anyio\abc\__init__.py", line 83, in <module>
    from ..from_thread import BlockingPortal
  File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\anyio\from_thread.py", line 4, in <module>
    from asyncio import iscoroutine
  File "\\v-tm-qeq-05\Prefect_Env\lib\asyncio\__init__.py", line 8, in <module>
    from .base_events import *
  File "\\v-tm-qeq-05\Prefect_Env\lib\asyncio\base_events.py", line 18, in <module>
    import concurrent.futures
  File "\\v-tm-qeq-05\Prefect_Env\lib\concurrent\futures\__init__.py", line 8, in <module>
    from concurrent.futures._base import (FIRST_COMPLETED,
  File "\\v-tm-qeq-05\Prefect_Env\lib\concurrent\futures\_base.py", line 7, in <module>
    import logging
  File "\\v-tm-qeq-05\PrefectScripts\cleung\Script\logging.py", line 1, in <module>
    from prefect import flow, task, get_run_logger
ImportError: cannot import name 'flow' from partially initialized module 'prefect' (most likely due to a circular import) (\\v-tm-qeq-05\Prefect_Env\lib\site-packages\prefect\__init__.py)
Any ideas?
I have a feeling its from teh circular import of the logging, but im not 100% sure how to solve
j

Jake Kaplan

08/23/2023, 2:06 PM
I would try and name 
logging.py
something else, i've seen that lead to circular import issues. Aside from that it is hard to say, it's very case by case for your own code 🙂
c

Charles Leung

08/23/2023, 5:23 PM
Ah, that was the issue - Thank you so much for your help !!
Powered by