https://prefect.io logo
Channels
Powered by
l

liren zhang

12/10/2020, 6:01 PM
Hi Community, I am trying to save the execution log to local file system in a hybrid prefect cloud with local runner mode with the following code
Copy code
import prefect
import prefect.utilities.context
from prefect.environments.storage import Local
from prefect.engine.results import LocalResult
from prefect import task, Flow

with prefect.context(a=1, b=2):
    print(prefect.context.a) # 1

@task(name='task_today')
def report_start_day():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>(prefect.context.today)
 
    <http://logger.info|logger.info>(prefect.context.flow_run_id)

with Flow('My flow', storage=Local(), result=LocalResult()) as flow:
	report_start_day()

flow.register(project_name="prefect_first_project")
z

Zanie

12/10/2020, 6:11 PM
Hi! The 
LocalResult
is for the return value from the task. The Prefect logger doesn’t write to a file by default, you can attach another stream handler to the logger that writes to a local file if you’d like the though
A rough example
Copy code
In [1]: from logging import getLogger, FileHandler

In [2]: logger = getLogger("prefect")

In [3]: logger.addHandler(FileHandler("foo.log"))

In [4]: import prefect
   ...: import prefect.utilities.context
   ...: from prefect.environments.storage import Local
   ...: from prefect.engine.results import LocalResult
   ...: from prefect import task, Flow
   ...: with prefect.context(a=1, b=2):
   ...:     print(prefect.context.a) # 1
   ...: @task(name='task_today')
   ...: def report_start_day():
   ...:     logger = prefect.context.get("logger")
   ...:     <http://logger.info|logger.info>(prefect.context.today)
   ...:     <http://logger.info|logger.info>(prefect.context.flow_run_id)
   ...: with Flow('My flow', storage=Local(), result=LocalResult()) as flow:
   ...: ^Ireport_start_day()
   ...: flow.run()
1
[2020-12-10 12:14:45] INFO - prefect.FlowRunner | Beginning Flow run for 'My flow'
[2020-12-10 12:14:45] INFO - prefect.TaskRunner | Task 'task_today': Starting task run...
[2020-12-10 12:14:45] INFO - prefect.task_today | 2020-12-10
[2020-12-10 12:14:45] INFO - prefect.task_today | ac1aff53-37c5-4a80-8044-b9009827cdbc
[2020-12-10 12:14:45] INFO - prefect.TaskRunner | Task 'task_today': Finished task run for task with final state: 'Success'
[2020-12-10 12:14:45] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
Out[5]: <Success: "All reference tasks succeeded.">

In [5]:                                                                                                                                                                                                                                       
Do you really want to exit ([y]/n)? 

❯ cat foo.log
Beginning Flow run for 'My flow'
Task 'task_today': Starting task run...
2020-12-10
ac1aff53-37c5-4a80-8044-b9009827cdbc
Task 'task_today': Finished task run for task with final state: 'Success'
Flow run SUCCESS: all reference tasks succeeded
l

liren zhang

12/10/2020, 7:22 PM
Hi @Zanie Thanks for sharing the sample code. when I execute the code as flow.run() the log did get save into foo.log. However, foo.log did not contain anything when I ran the flow from the cloud. My goal is to execute the flow in Prefect cloud and get a local copy of the log like the one we see in Prefect Cloud interface
z

Zanie

12/10/2020, 7:31 PM
You’ll probably need to call 
logger.addHandler(FileHandler("foo.log"))
from within your task
l

liren zhang

12/10/2020, 8:28 PM
@Zanie moving the log handler into task did not do me any good. I think I need to ask the question in the different way. DO you have a sample code to shows how to stream logs from Prefect back down to the local execution environment? I saw I could do the following in Prefect CLI 
prefect agent local start --show-flow-logs
and thought this is exactly what I need; but I will need to be able to do this in python
z

Zanie

12/10/2020, 9:29 PM
Copy code
❯ cat ~/prefect/scripts/example-log-locally.py
import os
from logging import FileHandler

import prefect
from prefect.environments.storage import Local
from prefect import task, Flow


LOG_PATH = os.path.expanduser("~/flow-that-logs.log")

def get_logger():
    logger = prefect.context.get("logger")
    logger.addHandler(FileHandler(LOG_PATH))
    return logger

@task()
def report_start_day():
    logger = get_logger()
    <http://logger.info|logger.info>(prefect.context.today)
    <http://logger.info|logger.info>(prefect.context.flow_run_id)


with Flow('flow-that-logs', storage=Local()) as flow:
	report_start_day()

flow.register(project_name="first")
Copy code
❯ python example-log-locally.py
Result check: OK
Flow URL: <https://cloud.prefect.io/michael-prefect-personal/flow/fdef6d85-4ebe-467f-bdcf-911ed50cfebc>
 └── ID: 9d8e4451-b6db-4726-b9d8-720b2c2d877c
 └── Project: first
 └── Labels: ['mzmbp.local']
Copy code
❯ prefect agent local start

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2020-12-10 21:22:47,646] INFO - agent | Starting LocalAgent with labels ['mzmbp.local', 'azure-flow-storage', 'gcs-flow-storage', 's3-flow-storage', 'github-flow-storage', 'webhook-flow-storage', 'gitlab-flow-storage']
[2020-12-10 21:22:47,646] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-12-10 21:22:47,646] INFO - agent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2020-12-10 21:22:47,939] INFO - agent | Waiting for flow runs...
[2020-12-10 21:23:06,457] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-12-10 21:23:06,905] INFO - agent | Deploying flow run 10377b6e-3d59-4447-8692-db101134ee12
Copy code
❯ cat ~/flow-that-logs.log
2020-12-10
10377b6e-3d59-4447-8692-db101134ee12
Is this the behavior you are looking for?
l

liren zhang

12/11/2020, 12:51 AM
It is something like this. but I am hoping for the full log which include system standard log messages in the file
Copy code
Beginning Flow run for 'My flow'
Using executor type LocalExecutor
Flow 'My flow': Handling state change from Scheduled to Running
Task 'task_today': Starting task run...
Task 'task_today': Handling state change from Pending to Running
Task 'task_today': Calling task.run() method...
2020-12-11
62f28a1c-822e-47b9-ad04-d97532090daa
Task 'task_today': Handling state change from Running to Success
Task 'task_today': Finished task run for task with final state: 'Success'
Flow run SUCCESS: all reference tasks succeeded
Flow 'My flow': Handling state change from Running to Success
Don't mind the formatting for now. I would like to see all the message above to be saved in the log file
z

Zanie

12/11/2020, 4:20 PM
I’m not sure there’s a great way to do this. Those logs are coming from the flow runner, not the flow.
l

liren zhang

12/11/2020, 5:25 PM
@Zanie you are exactly right. I do want to capture the logs from the flow runner. I am looking for a way to create a logger which inherits from Prefect that exposes the log outputs from flow runner/ task runner
z

Zanie

12/11/2020, 5:26 PM
You will not be able to create that logger from inside your flow because your flow is spawned by the flow runner.
You could open an issue requesting a feature to write prefect logs to a local file
l

liren zhang

12/11/2020, 5:47 PM
When I used flow.run() the Prefect logs do get captured with the details from the flowrunner. but when I register the same flow to cloud and execute it, the log does not get saved locally. However, I have my local agent run as 
prefect agent local start --show-flow-logs
and I do see the same log as I saw in the cloud interface. I am thinking if the agent can capture the logs and display them locally in my agent console, I should be able to get it into a log file
z

Zanie

12/11/2020, 5:48 PM
Yes, you’d have to either modify the agent code or pipe the output of the agent to a file
l

liren zhang

12/11/2020, 5:51 PM
would this help me in anyway? it says may libs set up to emit internal logs. How about prefect?
z

Zanie

12/11/2020, 5:52 PM
That pipes output from extra loggers into the prefect logger prefect cloud log handler
Powered by