Thread
#prefect-community
    d

    Darshan

    1 year ago
    Hello, If I am not using any backend but just a core library, is there a way to get a unique Id generated in flow context if I run a flow using "prefect run --path" command ?
    Michael Adkins

    Michael Adkins

    1 year ago
    We actually generate a UUID for the context even with the core library so that it feels similar to running it with a backend
    import prefect
    from prefect import task, Flow, Parameter
    
    
    @task(log_stdout=True)
    def say_hello(to: str) -> None:
        print(f"Hello {to} from flow run {prefect.context.flow_run_id}")
    
    
    @task()
    def capitalize(word: str) -> str:
        return word.capitalize()
    
    
    with Flow("hello-world") as hello_flow:
        name = Parameter("name", default="world")
        say_hello(capitalize(name))
    e.g.
    ❯ prefect run -p flow.py            
    Retrieving local flow... Done
    Running flow locally...
    └── 11:59:01 | INFO    | Beginning Flow run for 'hello-world'
    └── 11:59:01 | INFO    | Task 'name': Starting task run...
    └── 11:59:01 | INFO    | Task 'name': Finished task run for task with final state: 'Success'
    └── 11:59:01 | INFO    | Task 'capitalize': Starting task run...
    └── 11:59:01 | INFO    | Task 'capitalize': Finished task run for task with final state: 'Success'
    └── 11:59:01 | INFO    | Task 'say_hello': Starting task run...
    └── 11:59:01 | INFO    | Hello World from flow run 3ba3db36-0838-4b2d-a495-4f2507a675b1
    └── 11:59:01 | INFO    | Task 'say_hello': Finished task run for task with final state: 'Success'
    └── 11:59:01 | INFO    | Flow run SUCCESS: all reference tasks succeeded
    Flow run succeeded!
    d

    Darshan

    1 year ago
    Thanks for the examples, I did see flow_run_id available within tasks, but when I am using flow_run_id in log format then "Beginning Flow run for" log statement does not get it...but when I supply value to flow context manually instead of CLI then it works fine..
    Michael Adkins

    Michael Adkins

    1 year ago
    I see, so you're trying to insert the flow run id into the log formatter and running into something like this?
    ❯ python flow.py        
    --- Logging error ---
    Traceback (most recent call last):
      File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 436, in format
        return self._format(record)
      File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 432, in _format
        return self._fmt % record.__dict__
    KeyError: 'flow_run_id'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 1081, in emit
        msg = self.format(record)
      File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 925, in format
        return fmt.format(record)
      File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 667, in format
        s = self.formatMessage(record)
      File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 636, in formatMessage
        return self._style.format(record)
      File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 438, in format
        raise ValueError('Formatting field not found in record: %s' % e)
    ValueError: Formatting field not found in record: 'flow_run_id'
    Call stack:
      File "flow.py", line 19, in <module>
        hello_flow.run()
      File "/Users/mz/prefect/core/src/prefect/core/flow.py", line 1274, in run
        state = self._run(
      File "/Users/mz/prefect/core/src/prefect/core/flow.py", line 1092, in _run
        flow_state = runner.run(
      File "/Users/mz/prefect/core/src/prefect/engine/flow_runner.py", line 245, in run
        <http://self.logger.info|self.logger.info>("Beginning Flow run for '{}'".format(self.flow.name))
    Message: "Beginning Flow run for 'hello-world'"
    Arguments: ()
    [2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'name': Starting task run...
    [2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'name': Finished task run for task with final state: 'Success'
    [2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'capitalize': Starting task run...
    [2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'capitalize': Finished task run for task with final state: 'Success'
    [2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'say_hello': Starting task run...
    [2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Hello World from flow run 363b6cad-aa87-459c-9397-77c9e2f55a2c
    [2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'say_hello': Finished task run for task with final state: 'Success'
    [2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
    d

    Darshan

    1 year ago
    Exactly
    Michael Adkins

    Michael Adkins

    1 year ago
    I don't think we really intend for that to work, there are lots of logs that are written outside the context of a flow run and that formatter is for the
    prefect
    module logger
    It's not in context here because we display that log before actually starting the flow run and generating the proper context
    d

    Darshan

    1 year ago
    Ok I understand but if I want to have same format printed by app logger and prefect logger then only option I have is inject custom variables manually in the context before starting a flow ?
    Michael Adkins

    Michael Adkins

    1 year ago
    What do you mean by app logger?
    The CLI?
    d

    Darshan

    1 year ago
    No I meant logger created by my app to use outside of flow code in other python modules
    Michael Adkins

    Michael Adkins

    1 year ago
    I see. What format are you trying to use for your other logger? You can make them the same format you just can't include the flow run id in every prefect module log
    We're looking at rehauling our logging configuration options in the near future which would make this easier to setup per logger
    d

    Darshan

    1 year ago
    Basically I am looking to have a flow name and flow run id in every log statement within flow execution from both prefect core as well as my own python modules
    Michael Adkins

    Michael Adkins

    1 year ago
    Yeah I'm not sure we can currently support this since there are logs outside of a flow run context from the core module. At some point we'll let you set a different formatter for different prefect loggers so the flow run logger could have the format you want
    d

    Darshan

    1 year ago
    Yeah that would be ideal, thanks
    Michael Adkins

    Michael Adkins

    1 year ago
    https://github.com/PrefectHQ/prefect/pull/4550 is the PR to watch -- but it's low on my long list of things right now 🙂