Log Message Bifurcation for Storage Capacity and Access Time
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current log management tools face inefficiencies in storing and accessing archived log messages, as retrieving data from cold storage is time-consuming and requires manual intervention, leading to delays that can impact system performance and user experience.
Innovation Solution
The method of log message bifurcation, where each log message exceeding a storage limit is split into a log pattern and a metric object, with the log patterns stored in a log-pattern database and metric objects in a time-series metric database, allowing for efficient long-term storage and rapid access to archived messages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If log messages are stored in cold storage for long-term archiving, then storage cost is reduced, but access time increases significantly requiring manual re-ingestion
Solution Approach 1:
The patent segments log messages into two distinct components: log patterns (stored in log-pattern database) and metric objects (stored in time-series metric database). This segmentation allows each component to be optimized independently - patterns provide structural consistency while metric objects capture variable data, enabling efficient retrieval without manual re-ingestion of entire log messages.
Solution Approach 2:
The system performs preliminary processing by bifurcating log messages into patterns and metric objects during the archiving phase. This preliminary action structures the data in advance, so when access is needed, the components can be quickly reassembled without requiring time-consuming manual re-ingestion processes.
2Quantity of substance
If conventional cold storage is used for archiving log messages, then storage efficiency improves, but retrieval process becomes complex requiring manual intervention
Solution Approach 1:
By dividing log messages into log patterns and metric objects stored in separate databases, the system simplifies the retrieval process. Each database can be queried independently using optimized methods, and the results combined automatically, eliminating the need for complex manual re-ingestion procedures.
Solution Approach 2:
The patent introduces an intermediary reconstruction process that automatically combines log patterns and metric objects into complete log messages. This intermediary mechanism handles the complexity of retrieval internally, presenting a simple interface to users while managing the sophisticated data structure behind the scenes.
3Quantity of substance
If archived log messages are stored in cold storage, then storage cost is reduced, but system performance deteriorates due to delays in accessing historical data
Solution Approach 1:
The segmentation of log data into patterns and metric objects enables parallel query processing and efficient data retrieval. This structure allows the system to maintain high productivity when accessing archived logs, as the segmented data can be quickly reconstructed without the performance penalties of traditional cold storage retrieval.
Data Source
AI summary
Methods and systems described herein are directed to storing and querying log messages using log message bifurcation. Each log message that exceeds a time limit for storage in a live storage database is bifurcated into a log pattern that is stored in a log-pattern database and a metric object that records variable information of the log message in a time-series metric database. The log-pattern database and the time-series metric database together provide long-term log storage for the information contained in log messages that are too old to be stored in a live storage database. A request for access to log messages stored in the long-term log storage is completed by reconstructing the requested log messages from log patterns stored in the log-pattern database and metric objects stored in the time-series metric database.