Stratified Sampling of Log Records for Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently storing and searching large volumes of log records generated by data centers, as the storage requirements and search times are proportional to the number of log records, leading to excessive resource usage and slow query performance.
Innovation Solution
A method that employs stratified sampling to index log records based on template identifiers and terms, where sampling count thresholds determine the indexing rate, allowing for efficient storage and search of frequently occurring combinations while prioritizing infrequent occurrences for higher fidelity analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all log records are stored in a full-text index, then complete search capability is achieved, but storage requirements become excessive and search time increases
Solution Approach 1:
The patent segments the log record population into different strata based on template identifiers and terms. Each stratum is sampled at different rates, with frequent templates sampled less densely and infrequent templates sampled more densely or fully. This segmentation allows the system to maintain search completeness for rare events while reducing storage for common events.
Solution Approach 2:
Different sampling rates are applied to different segments (strata) of log records based on their frequency characteristics. Instead of uniform sampling, the system applies local quality control by adjusting sampling intensity according to the specific template-term combination, ensuring optimal representation for each category.
2Measurement precision
If all log records are indexed, then search accuracy is maximized, but search time and processing overhead increase
Solution Approach 1:
The sampling rate is made dynamic and adaptive rather than static. The system continuously monitors template-term frequency and adjusts sampling rates accordingly. This dynamic approach allows the index to adapt to changing log patterns, maintaining search accuracy for emerging rare events while optimizing performance for common events.
Solution Approach 2:
The system changes the sampling parameter (sampling rate) based on the frequency characteristics of different template-term combinations. By adjusting this parameter dynamically, the system optimizes the balance between search accuracy and search time for different categories of log records.
3Quantity of substance
If sampling rate is increased for frequent log records, then storage efficiency improves, but search fidelity for rare events deteriorates
Solution Approach 1:
The system applies different sampling qualities to different segments of log records. Frequent template-term combinations are sampled at lower rates (higher efficiency), while rare combinations are sampled at higher rates or fully (higher fidelity). This local quality differentiation resolves the contradiction by optimizing for the right metric in the right context.
Solution Approach 2:
By segmenting log records into strata based on frequency, the system can apply appropriate sampling strategies to each segment. Rare events are preserved with high fidelity in their own stratum, while common events are efficiently sampled, allowing the overall system to achieve both storage efficiency and search fidelity.
4Device complexity
If uniform sampling is applied to all log records, then implementation simplicity is maintained, but resource utilization becomes inefficient
Solution Approach 1:
The system segments log records by template identifier and term to create frequency-based strata. This segmentation enables differential sampling strategies that improve resource utilization while adding manageable complexity through the use of pre-computed frequency statistics and structured sampling logic.
Data Source
AI summary
A log record from a host machine node includes an invariant string and a term. A template identifier is selected, from among template identifiers within a template repository, for a template string matching the invariant string. A sampling count threshold is selected from among a set of sampling count thresholds based on the template identifier and the term. A template-term count is obtained based on a number of earlier log records that were received since the count was reset and have a template identifier and a term that match the template identifier and the term of the log record. Based on the template-term count satisfying the sampling count threshold, an index entry is generated in a sampled log records index based on the log record and the template-term count is reset to a defined value. Based on the template-term count not satisfying the sampling count threshold, the template-term count is incremented.