Message Extraction Using Format and Keyword Weight Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information processing systems face challenges in extracting desired messages from log files and other messages when software updates change message formats, such as changes in spacing, delimiters, or word order, leading to missed desired messages due to strict format definitions.
Innovation Solution
An information processing apparatus that stores first filter information specifying message formats and second filter information specifying weights for words or phrases, allowing it to select messages matching the formats and calculate importance levels based on included words or phrases, enabling flexible extraction of messages with changed formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If strict format definitions are used in filter information to reduce the possibility of extracting unwanted messages, then extraction precision is improved, but adaptability deteriorates because desired messages may not be extracted when software formats change
Solution Approach 1:
The filter information is segmented into two independent parts: format specifications (first filter information) and keyword weights (second filter information). This segmentation allows the system to separately handle format matching and semantic evaluation, enabling flexible adaptation to format changes while maintaining extraction precision through the keyword weight component.
Solution Approach 2:
The system changes the parameter representation from strict format strings to weighted keyword scores. By converting the format definition into a flexible scoring mechanism based on keyword importance weights, the system can adapt to various message formats while maintaining the ability to distinguish desired messages through the weighted evaluation of key terms.
2Adaptability or versatility
If filter information is updated frequently to reflect format changes, then adaptability is improved, but device complexity increases due to the need for continuous filter maintenance
Solution Approach 1:
The system implements self-service through automatic filter information generation. The message extraction apparatus can automatically create or update filter information based on analyzed messages, eliminating the need for manual filter maintenance and reducing device complexity while maintaining high adaptability to format changes.
Solution Approach 2:
The system performs preliminary analysis of message formats and contents to pre-generate or pre-update filter information. By anticipating format changes through preliminary analysis of incoming messages, the system can proactively update its extraction criteria without requiring complex real-time adaptation mechanisms.
3Speed
If only format-based filtering is used, then processing speed is improved through simple matching, but measurement precision deteriorates because messages with changed formats are missed
Solution Approach 1:
The filtering process is segmented into two stages: rapid format-based initial filtering using the first filter information, followed by precision keyword-based evaluation using the second filter information. This segmentation maintains processing speed through the fast initial filter while improving extraction accuracy through the subsequent keyword weight evaluation of messages that pass the initial filter.
Solution Approach 2:
The system applies partial format matching rather than strict format enforcement. By accepting messages that partially match the expected format and then evaluating them with keyword weights, the system processes more messages through the initial filter (excessive action) but achieves higher precision through the selective keyword evaluation of the subset that passes.
Data Source
AI summary
A storage unit stores first filter information specifying the formats of messages and second filter information specifying weights for words or phrases. A first search unit selects messages matching the formats specified by the first filter information from a plurality of messages as messages to be extracted. A second search unit calculates the importance level of each message unselected by the first search unit, based on the words or phrases included in the message and the second filter information, and selects messages to be extracted, according to the calculated importance levels from the messages unselected by the first search unit.