< Marvin> show me a prefect yml that sets PREFECT LOGGING LO Prefect Community #ask-marvin

Join Slack

<@ULVA73B9P> show me a prefect.yml that sets PREFE...

# ask-marvin

Ivo

10/19/2024, 11:51 AM

@Marvin show me a prefect.yml that sets PREFECT_LOGGING_LOG_PRINTS to true with prefect v3.0.10

Marvin

10/19/2024, 11:52 AM

To set the

PREFECT_LOGGING_LOG_PRINTS

in Prefect v3.0.10, you can configure it through the

prefect.yaml

file or directly in your flow configuration. Although the documentation did not provide a direct example with

PREFECT_LOGGING_LOG_PRINTS

prefect.yaml

, you can usually set environment variables or configuration options in Prefect through such a file by following the infrastructure and deployment configuration practices. Here's an indicative approach on how you might achieve this: 1. prefect.yaml File: If you have a

prefect.yaml

file, you can specify environment variables for your deployments:

Copy code

yaml
   version: 2
   description: A description of your deployment
   name: my-deployment
   script_path: my_flow.py

   infrastructure:
     env:
       PREFECT_LOGGING_LOG_PRINTS: 'true'

2. Flow Configuration: Alternatively, you can explicitly define

log_prints=True

within your flow configuration. Here's a basic example:

Copy code

python
   from prefect import flow

   @flow(log_prints=True)
   def my_flow():
       print("This will be logged")

   if __name__ == "__main__":
       my_flow()

For more details, you can refer to the Prefect v3 documentation on logging and how to define deployments with YAML. If you encounter any unexpected issues or behaviors, it's also a good idea to check the release notes for specific changes or bug fixes that might affect your setup.

Ivo

10/19/2024, 11:58 AM

single quotes, that was it, thank you! Everything else throws a schema error

Ivo

10/19/2024, 1:49 PM

@Marvin is this logging.yml properly formatted?

Copy code

# Prefect logging config file.
#
# Any item in this file can be overridden with an environment variable:
#    `PREFECT_LOGGING_[PATH]_[TO]_[KEY]=VALUE`
#
# Templated values can be used to insert values from the Prefect settings at runtime.

version: 1
disable_existing_loggers: False

formatters:
    simple:
        format: "%(asctime)s.%(msecs)03d | %(message)s"
        datefmt: "%H:%M:%S"

    standard:
        (): prefect.logging.formatters.PrefectFormatter
        format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | %(name)s - %(message)s"
        flow_run_fmt: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Flow run %(flow_run_name)r - %(message)s"
        task_run_fmt: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Task run %(task_run_name)r - %(message)s"
        datefmt: "%H:%M:%S"

    debug:
        format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | %(threadName)-12s | %(name)s - %(message)s"
        datefmt: "%H:%M:%S"

    slack:
        (): prefect.logging.formatters.PrefectFormatter
        format: "%(asctime)s.%(msecs)03d | %(message)s"
        flow_run_fmt: "%(asctime)s.%(msecs)03d | Flow run %(flow_run_name)r - %(message)s"
        task_run_fmt: "%(asctime)s.%(msecs)03d | Task run %(task_run_name)r - %(message)s"
        datefmt: "%H:%M:%S"

    json:
        class: prefect.logging.formatters.JsonFormatter
        format: "default"

# filters:
    # Define any custom filters to drops records containing
    # sensitive information
    # my_filter:
        # class: your_module.FilterClass

handlers:

    # The handlers we define here will output all logs they receive by default
    # but we include the `level` so it can be overridden by environment

    console:
        level: 0
        class: prefect.logging.handlers.PrefectConsoleHandler
        formatter: standard
        styles:
            log.web_url: bright_blue
            log.local_url: bright_blue

            log.info_level: cyan
            log.warning_level: yellow3
            log.error_level: red3
            log.critical_level: bright_red

            log.completed_state: green
            log.cancelled_state: yellow3
            log.failed_state: red3
            log.crashed_state: bright_red

            log.flow_run_name: magenta
            log.flow_name: bold magenta

    api:
        level: 0
        class: prefect.logging.handlers.APILogHandler

    
    debug:
        level: 0
        class: logging.StreamHandler
        formatter: debug

    slack:
        level: 40
        class: flows.SlackErrorReportHandler
        formatter: slack

    

loggers:
    prefect:
        level: "${PREFECT_LOGGING_LEVEL}"

    prefect.extra:
        level: "${PREFECT_LOGGING_LEVEL}"
        handlers: [api]

    prefect.flow_runs:
        level: NOTSET
        handlers: [api, slack]

    prefect.task_runs:
        level: NOTSET
        handlers: [api, slack]

    prefect.server:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"

    prefect.client:
        level: "${PREFECT_LOGGING_LEVEL}"

    prefect.infrastructure:
        level: "${PREFECT_LOGGING_LEVEL}"

    prefect._internal:
        level: "${PREFECT_LOGGING_INTERNAL_LEVEL}"
        propagate: false
        handlers: [debug]

    uvicorn:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"

    fastapi:
        level: "${PREFECT_LOGGING_SERVER_LEVEL}"

# The root logger: any logger without propagation disabled sends to here as well
root:
    # By default, we display warning level logs from any library in the console
    # to match Python's default behavior while formatting logs nicely
    level: WARNING
    handlers: [console]

Marvin

10/19/2024, 1:49 PM

Your message was too long by 632 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

is this logging.yml properly formatted?

```# Prefect logging config file. # # Any item in this file can be overridden with an environment variable: #

PREFECT_LOGGING_[PATH]_[TO]_[KEY]=VALUE

# # Templated values can be used to insert values from the Prefect settings at runtime. version: 1 disable_existing_loggers: False formatters: simple: format: "%(asctime)s.%(msecs)03d | %(message)s" datefmt: "%H:%M:%S" standard: (): prefect.logging.formatters.PrefectFormatter format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | %(name)s - %(message)s" flow_run_fmt: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Flow run %(flow_run_name)r - %(message)s" task_run_fmt: "%(asctime)s.%(msecs)03d | %(levelname)-7s | Task run %(task_run_name)r - %(message)s" datefmt: "%H:%M:%S" debug: format: "%(asctime)s.%(msecs)03d | %(levelname)-7s | %(threadName)-12s | %(name)s - %(message)s" datefmt: "%H:%M:%

🙏 1

Ivo

10/19/2024, 4:31 PM

Hi @Marvin, can you show a sample directory structure for a project with a custom handler class referenced in logging.yml

Marvin

10/19/2024, 4:32 PM

To set up a project with a custom handler class referenced in

logging.yml

, you can follow a directory structure like this:

Copy code

my_prefect_project/
│
├── flows/
│   └── my_flow.py
│
├── handlers/
│   └── custom_handler.py
│
├── configs/
│   └── logging.yml
│
└── prefect.yaml

- my_prefect_project: The root directory for your project. - flows/: Directory containing your Prefect flow files, such as

my_flow.py

. - handlers/: Directory for custom utilities like logging handlers. This could include

custom_handler.py

, where you define your custom logging class. - configs/: A directory for configuration files, including

logging.yml

, where you specify your logging configuration. Example of a basic `logging.yml`:

Copy code

yaml
  version: 1
  formatters:
    simple:
      format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

  handlers:
    custom:
      class: handlers.custom_handler.CustomHandlerClass
      formatter: simple

  root:
    level: INFO
    handlers: [custom]

- prefect.yaml: Configuration file for Prefect, setting up deployments or additional configurations. Ensure that

handlers/custom_handler.py

includes the definition for

CustomHandlerClass

that extends or implements Python's logging functionality. This setup allows you to maintain a clean structure, separating handlers and flows while making it easier to modify or update specific components independently. For more insights, you can review Prefect's logging documentation.

Ivo

10/19/2024, 4:45 PM

@Marvin what does the entrypoint for the prefect.yml deployment look like in that example?

Marvin

10/19/2024, 4:46 PM

In the

prefect.yaml

file used for defining a deployment in the example directory structure, the

entrypoint

would specify the path to the flow script that needs to be executed. Given the example structure, the

entrypoint

would refer to the

my_flow.py

file located in the

flows

directory. Here's how your

prefect.yaml

might look:

Copy code

yaml
version: 2
description: A description of your Prefect deployment
name: my-deployment
entrypoint: flows/my_flow.py:my_flow_function

In this example: - entrypoint: Indicates the path to the Python file containing the flow, followed by a colon and the function name that defines the flow (e.g.,

my_flow_function

). Make sure to replace

my_flow_function

with the actual name of the flow function defined in

my_flow.py

. This setup allows Prefect to know where to find and how to execute the flow function when the deployment is triggered. The

entrypoint

parameter is crucial for locating the correct flow entry point in your project.

2 Views

Open in Slack

Previous Next