Log Record Partitioning via Term Frequency and Bloom Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently storing and searching large volumes of log records generated by data centers, as full-text indexes require significant storage space and processing resources, making it time-consuming and resource-intensive to manage and query these records.
Innovation Solution
The method involves partitioning log records based on term frequency and type, using Bloom filter vectors to associate identifiers with frequency and type partitions, allowing for efficient storage and skipping of non-relevant partitions during searches, thereby reducing processing and storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If full-text indexes are used to store log records, then complex text queries can be performed, but storage requirements and processing resources increase significantly
Solution Approach 1:
The patent segments log records into different partitions based on frequency thresholds. Frequently occurring log records are stored in a compressed format with shared common substrings, while infrequently occurring records are stored in a more detailed format. This segmentation allows the system to optimize storage space for the majority of records while preserving query capability for all record types.
Solution Approach 2:
The patent changes the storage parameter (compression level) based on the frequency parameter of log records. High-frequency records use higher compression with shared substrings, while low-frequency records use lower compression. This dynamic parameter adjustment resolves the contradiction by adapting storage efficiency to the actual query patterns.
2Measurement precision
If all log records are stored in detail, then search accuracy is maintained, but processing time and resource consumption increase
Solution Approach 1:
The patent segments the search process into two phases: first, a quick filter phase using compressed representations and frequency thresholds to eliminate irrelevant partitions, and second, a detailed verification phase only for promising candidates. This segmentation dramatically reduces search time while maintaining accuracy by avoiding detailed processing of obviously irrelevant records.
Solution Approach 2:
The patent implements skipping mechanisms where entire partitions of log records can be skipped during search based on frequency thresholds and pattern matching. If a partition's frequency characteristics don't match the query pattern, the system rushes through (skips) that partition entirely, avoiding unnecessary processing time while preserving the ability to find matches in relevant partitions.
3Quantity of substance
If log records are compressed to save space, then storage efficiency improves, but query processing complexity increases
Solution Approach 1:
The patent performs preliminary actions during the log record ingestion phase by pre-computing frequency statistics, identifying common substrings, and organizing records into frequency-based partitions. This preliminary organization creates an optimized structure that simplifies subsequent query processing, as the heavy lifting of compression and organization is done once during ingestion rather than repeatedly during queries.
Solution Approach 2:
The patent introduces frequency thresholds and partition metadata as intermediary structures between the compressed log records and the query processing logic. These intermediaries provide a simplified interface for queries to interact with the compressed data, translating high-level query requirements into efficient access patterns without requiring complex direct manipulation of compressed records.
Data Source
AI summary
A log record from a host machine node includes terms. Frequency of occurrence of the terms across a stream of log records is determined. Based on the frequency satisfying a threshold, a Bloom filter vector is selected from among a plurality of Bloom filter vectors based on the frequency, the Bloom filter vector is updated based on the terms, and an identifier for the log record is stored with an association to the Bloom filter vector. In contrast, based on the frequency of occurrence not satisfying the defined frequency range, a type identifier is identified based on the terms, a Bloom filter vector is selected from among the plurality of Bloom filter vectors based on the type identifier, the Bloom filter vector is updated based on the terms, and an identifier for the log record is stored with an association to the Bloom filter vector.