https://prefect.io logo
#prefect-community
Title
# prefect-community
p

Patrick Tan

03/25/2022, 7:28 PM
Hi, regarding prefect logger, is there any way to store them to external storage like my own database, in additions to Prefect Cloud? Eg [2022-03-25 151344-0400] INFO - prefect.TaskRunner | Task 'get_config['textlist_id']': Finished task run for task with final state: 'Success' [2022-03-25 151344-0400] INFO - prefect.TaskRunner | Task 'query_from_opensearch': Starting task run... [2022-03-25 151344-0400] INFO - prefect.TaskRunner | Task 'query_from_opensearch': Finished task run for task with final state: 'Success' [2022-03-25 151344-0400] INFO - prefect.TaskRunner | Task 'from_s3': Starting task run... [2022-03-25 151345-0400] INFO - prefect | Read from S3: 3 [2022-03-25 151345-0400] INFO - prefect.TaskRunner | Task 'from_s3': Finished task run for task with final state: 'Success' [2022-03-25 151345-0400] INFO - prefect.TaskRunner | Task 'validate': Starting task run...
k

Kevin Kho

03/25/2022, 7:29 PM
Basically you need to attach a handler like this example to set it where you want. This example is for local files
p

Patrick Tan

03/25/2022, 8:15 PM
This example store application log to local file, how about those log generated by prefect such as prefect.FlowRunner or prefect.TaskRunner? [2022-03-25 161318-0400] INFO - prefect.FlowRunner | Beginning Flow run for 'livelots flow' [2022-03-25 161318-0400] INFO - prefect.TaskRunner | Task 'get_config': Starting task run... [2022-03-25 161319-0400] INFO - prefect.TaskRunner | Task 'get_config': Finished task run for task with final state: 'Success'
k

Kevin Kho

03/25/2022, 8:37 PM
I believe if you use a Script Based storage, you can add the handler at the start of the Flow and everything will be logged together
p

Patrick Tan

03/27/2022, 2:14 PM
Thanks, will look into it
3 Views