amulog: A general log analysis framework for comparison and combination of diverse template generation methods

Satoru Kobayashi, Yuya Yamashiro, Kazuki Otomo and Kensuke Fukuda

International Journal of Network Management, 2022, vol. 32, issue 4

Abstract: One of the ways to analyze unstructured log messages from large‐scale IT systems is to classify log messages with log templates generated by template generation methods. However, there is currently no common knowledge pertained to the comparison and practical use of log template generation methods because they are implemented on the basis of diverse environments. To this end, we design and implement amulog, a general log analysis framework for comparing and combining diverse log template generation methods. Amulog consists of three key functions: (1) parsing log messages into headers and segmented messages, (2) classifying the log messages using a scalable template‐matching method, and (3) storing the structured data in a database. This framework helps us easily utilize time‐series data corresponding to the log templates for further analysis. We evaluate amulog with a log dataset collected from a nation‐wide academic network and demonstrate that it classifies the log data in a reasonable amount of time even with over 100,000 log template candidates. The template‐matching method in amulog also reduces 75% processing time for template generation and keeps the accuracy when combined with an existing structure‐based template generation method. In order to show the effectiveness of amulog in comparing log template generation methods, we demonstrate that the appropriate template generation methods and accuracy metrics largely depend on the purpose of further analysis by comparing the accuracy of six existing log template generation methods with 10 different accuracy metrics on amulog.

Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1002/nem.2195

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:wly:intnem:v:32:y:2022:i:4:n:e2195

Access Statistics for this article

More articles in International Journal of Network Management from John Wiley & Sons
Bibliographic data for series maintained by Wiley Content Delivery ().