Hello If I am not using any backend but just a core library Prefect Community #ask-community

Hello, If I am not using any backend but just a co...

Darshan

07/02/2021, 4:49 PM

Hello, If I am not using any backend but just a core library, is there a way to get a unique Id generated in flow context if I run a flow using "prefect run --path" command ?

Zanie

07/02/2021, 4:57 PM

We actually generate a UUID for the context even with the core library so that it feels similar to running it with a backend

upvote 1

Zanie

07/02/2021, 4:59 PM

e.g.

Copy code

❯ prefect run -p flow.py            
Retrieving local flow... Done
Running flow locally...
└── 11:59:01 | INFO    | Beginning Flow run for 'hello-world'
└── 11:59:01 | INFO    | Task 'name': Starting task run...
└── 11:59:01 | INFO    | Task 'name': Finished task run for task with final state: 'Success'
└── 11:59:01 | INFO    | Task 'capitalize': Starting task run...
└── 11:59:01 | INFO    | Task 'capitalize': Finished task run for task with final state: 'Success'
└── 11:59:01 | INFO    | Task 'say_hello': Starting task run...
└── 11:59:01 | INFO    | Hello World from flow run 3ba3db36-0838-4b2d-a495-4f2507a675b1
└── 11:59:01 | INFO    | Task 'say_hello': Finished task run for task with final state: 'Success'
└── 11:59:01 | INFO    | Flow run SUCCESS: all reference tasks succeeded
Flow run succeeded!

Zanie

07/02/2021, 4:59 PM

Copy code

import prefect
from prefect import task, Flow, Parameter


@task(log_stdout=True)
def say_hello(to: str) -> None:
    print(f"Hello {to} from flow run {prefect.context.flow_run_id}")


@task()
def capitalize(word: str) -> str:
    return word.capitalize()


with Flow("hello-world") as hello_flow:
    name = Parameter("name", default="world")
    say_hello(capitalize(name))

Darshan

07/02/2021, 5:07 PM

Thanks for the examples, I did see flow_run_id available within tasks, but when I am using flow_run_id in log format then "Beginning Flow run for" log statement does not get it...but when I supply value to flow context manually instead of CLI then it works fine..

Zanie

07/02/2021, 5:23 PM

I see, so you're trying to insert the flow run id into the log formatter and running into something like this?

Copy code

❯ python flow.py        
--- Logging error ---
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 436, in format
    return self._format(record)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 432, in _format
    return self._fmt % record.__dict__
KeyError: 'flow_run_id'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 1081, in emit
    msg = self.format(record)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 925, in format
    return fmt.format(record)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 667, in format
    s = self.formatMessage(record)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 636, in formatMessage
    return self._style.format(record)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect-dev-38/lib/python3.8/logging/__init__.py", line 438, in format
    raise ValueError('Formatting field not found in record: %s' % e)
ValueError: Formatting field not found in record: 'flow_run_id'
Call stack:
  File "flow.py", line 19, in <module>
    hello_flow.run()
  File "/Users/mz/prefect/core/src/prefect/core/flow.py", line 1274, in run
    state = self._run(
  File "/Users/mz/prefect/core/src/prefect/core/flow.py", line 1092, in _run
    flow_state = runner.run(
  File "/Users/mz/prefect/core/src/prefect/engine/flow_runner.py", line 245, in run
    <http://self.logger.info|self.logger.info>("Beginning Flow run for '{}'".format(self.flow.name))
Message: "Beginning Flow run for 'hello-world'"
Arguments: ()
[2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'name': Starting task run...
[2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'name': Finished task run for task with final state: 'Success'
[2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'capitalize': Starting task run...
[2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'capitalize': Finished task run for task with final state: 'Success'
[2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'say_hello': Starting task run...
[2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Hello World from flow run 363b6cad-aa87-459c-9397-77c9e2f55a2c
[2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.TaskRunner | Task 'say_hello': Finished task run for task with final state: 'Success'
[2021-07-02 12:22:58-0500] INFO - 363b6cad-aa87-459c-9397-77c9e2f55a2c prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded

Darshan

07/02/2021, 5:24 PM

Exactly

Zanie

07/02/2021, 5:25 PM

I don't think we really intend for that to work, there are lots of logs that are written outside the context of a flow run and that formatter is for the

prefect

module logger

Zanie

07/02/2021, 5:25 PM

It's not in context here because we display that log before actually starting the flow run and generating the proper context

Darshan

07/02/2021, 5:36 PM

Ok I understand but if I want to have same format printed by app logger and prefect logger then only option I have is inject custom variables manually in the context before starting a flow ?

Zanie

07/02/2021, 5:37 PM

What do you mean by app logger?

Zanie

07/02/2021, 5:38 PM

The CLI?

Darshan

07/02/2021, 5:40 PM

No I meant logger created by my app to use outside of flow code in other python modules

Zanie

07/02/2021, 5:43 PM

I see. What format are you trying to use for your other logger? You can make them the same format you just can't include the flow run id in every prefect module log

Zanie

07/02/2021, 5:44 PM

We're looking at rehauling our logging configuration options in the near future which would make this easier to setup per logger

👍 1

Darshan

07/02/2021, 5:48 PM

Basically I am looking to have a flow name and flow run id in every log statement within flow execution from both prefect core as well as my own python modules

Zanie

07/02/2021, 9:15 PM

Yeah I'm not sure we can currently support this since there are logs outside of a flow run context from the core module. At some point we'll let you set a different formatter for different prefect loggers so the flow run logger could have the format you want

Darshan

07/02/2021, 10:25 PM

Yeah that would be ideal, thanks

Zanie

07/02/2021, 10:30 PM

https://github.com/PrefectHQ/prefect/pull/4550 is the PR to watch -- but it's low on my long list of things right now 🙂

3 Views

Open in Slack

Previous Next