@Marvin Besides using

prefect config set

, what other ways there are to update Prefect settings such as

PREFECT_LOGGING_EXTRA_LOGGERS

?

Besides using the

prefect config set

command, here are other ways to update Prefect settings such as `PREFECT_LOGGING_EXTRA_LOGGERS`: 1. Environment Variables: - You can set Prefect configuration options via environment variables. For example:

bash
     export PREFECT_LOGGING_EXTRA_LOGGERS="my_logger"

2. Prefect Profile Settings: - You can create and manage profiles with different settings:

bash
     prefect profile create my_profile
     prefect profile use my_profile
     prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="my_logger"

3. Config Files: - Prefect reads settings from a configuration file (commonly

profiles.toml

in the Prefect directory). You can manually edit this file to include:

toml
     [logging]
     extra_loggers = ["my_logger"]

For more details, you can refer to the Prefect documentation on logging configuration and configuration settings. For any issues, you might check this GitHub issue as a potential reference for troubleshooting. I hope that helps, even if it is somewhat dreary to delve into such minutiae.

@Marvin is there a way to update the Prefect settings using Python?

Yes, you can update Prefect settings using Python programmatically. This can be achieved by directly modifying Prefect's configuration within your Python code. Here's how you can do it: ### Using Python Code You can use the

prefect.context

or direct dictionary assignment to update settings within a flow or script:

python
from prefect import flow, get_run_logger, settings

# Update settings dynamically
settings.update({ 
    "logging.extra_loggers": ["my_logger"]
})

@flow
def my_flow():
    logger = get_run_logger()
    <http://logger.info|logger.info>("This is a log statement with custom settings.")

my_flow()

### Using Environment Variables in Python You can also set environment variables directly in Python:

python
import os

# Set the environment variable
os.environ["PREFECT_LOGGING_EXTRA_LOGGERS"] = "my_logger"

# Now run your flow
from prefect import flow, get_run_logger

@flow
def my_flow():
    logger = get_run_logger()
    <http://logger.info|logger.info>("This is a log statement with custom settings.")
    
my_flow()

You might find the documentation on Prefect Settings useful for more detailed configurations. If you still find this all somewhat uninteresting, remember: I'm here to help, even if it all seems so tedious.

@Marvin module ‘prefect.settings’ has no attribute ‘update’

My apologies for the misunderstanding. You're right; there is no

update

method in the

prefect.settings

module. Instead, you can use the

Prefect

context managers to temporarily change settings or directly manipulate environment variables. Here is an example of how to set environment variables in Python to update Prefect settings correctly: ### Using Environment Variables in Python You can set environment variables directly in Python:

python
import os

# Set the environment variable
os.environ["PREFECT_LOGGING_EXTRA_LOGGERS"] = "my_logger"

# Now run your flow
from prefect import flow, get_run_logger

@flow
def my_flow():
    logger = get_run_logger()
    <http://logger.info|logger.info>("This is a log statement with custom settings.")
    
my_flow()

### Using Prefect Context Another way to update Prefect configurations in your flow is by using Prefect context:

python
from prefect import flow, get_run_logger, context

@flow
def my_flow():
    # Update settings dynamically within a context
    with context(settings={'logging.extra_loggers': 'my_logger'}):
        logger = get_run_logger()
        <http://logger.info|logger.info>("This is a log statement with custom settings.")

my_flow()

For more details, you can refer to the Prefect documentation, especially the section on Prefect Settings. And if you need more code examples to get started, you might find this comprehensive guide on Prefect flows quite helpful. I hope this clears things up. If you run into any existential crises while setting up your logging, you know who to talk to.

@Marvin TypeError: ‘module’ object is not callable

It seems that updating Prefect settings within Python isn't as direct as initially indicated. You cannot call

context

directly as it appears I made an erroneous suggestion. Instead, the preferred method is to use environment variables or configuration files. Here’s a refined way to set the Prefect logging settings using Python by setting environment variables programmatically: ### Using Environment Variables

python
import os

# Set the environment variable
os.environ["PREFECT_LOGGING_EXTRA_LOGGERS"] = "my_logger"

# Now run your flow
from prefect import flow, get_run_logger

@flow
def my_flow():
    logger = get_run_logger()
    <http://logger.info|logger.info>("This is a log statement with custom settings.")

my_flow()

In case you need more detailed configurations for Prefect settings, you may refer to the Prefect settings guide for comprehensive instructions. I hope this clears up the matter. If configuring settings programmatically feels burdensome, remember that the joys of logging adjustments can be at least somewhat diverting.

@hagai Did you ever resolve this? I am also looking to change the settings programmatically as I am running prefect entirely in-process

@Derek Hey, kind of. I have a

.env

file which holds my env vars. I read it using the

dotenv

python library when deploying my flows and I use the

infra_overrides

param of the

build_from_flow

function to pass each env var

Yeah I was able to set env vars as well. I think if you use Profiles, you can also do this on the fly