Probabilistic Log Indexing for Fast Queries and Compact Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems struggle with inefficient storage and querying of massive log data, leading to high CPU usage and long query times, especially when searching for keywords in large datasets, and existing solutions like Elastic/Lucene DBMS face challenges with indexing tradeoffs and data size issues.
Innovation Solution
A probabilistic data structure is constructed using three tables: Token Map, Entities List, and Lookup Map, employing hash values and commutative hash functions to store and query log lines efficiently, allowing for compact data storage and fast querying speeds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional database indexing is used to store and query log lines, then query performance is improved, but storage size increases significantly (index larger than actual data)
Solution Approach 1:
The patent creates a probabilistic copy ( Bloom filter structure) of the log line data that enables fast query operations without storing the complete original data. The Bloom filter contains hash values that represent the log lines, allowing O(1) query time while occupying minimal space compared to traditional indexes.
Solution Approach 2:
The patent transforms the exact matching problem into a probabilistic membership test by changing the data representation from actual log lines to hash values. This parameter transformation enables compact storage while maintaining query functionality through hash-based lookups.
2Duration of action of stationary object
If massive amounts of log data are stored over long periods, then data retention is improved, but query performance degrades due to data volume
Solution Approach 1:
The patent divides the log data into batches and creates separate Bloom filters for each batch. This segmentation allows the system to retain massive amounts of data over long periods while maintaining fast query performance by only searching relevant batches rather than the entire dataset.
Solution Approach 2:
The patent adds a temporal dimension to the storage structure by organizing data in time-based batches with hierarchical indexing. This enables efficient queries over long retention periods by navigating through time dimensions rather than scanning all data linearly.
3Measurement precision
If full indexing is performed on all log data, then query accuracy is improved, but CPU usage increases to unacceptable levels
Solution Approach 1:
The patent applies partial indexing by creating Bloom filters only for specific batches or frequently queried data rather than indexing all log data. This partial action reduces CPU usage significantly while maintaining sufficient query accuracy for the most important data.
Solution Approach 2:
The patent uses computationally inexpensive hash functions to create Bloom filters that can be quickly generated and discarded. These probabilistic structures require minimal CPU resources compared to traditional indexes, enabling the system to handle massive data volumes with acceptable energy consumption.
4Quantity of substance
If data is stored in compressed form to reduce size, then storage efficiency is improved, but query processing time increases due to decompression requirements
Solution Approach 1:
The patent pre-computes and stores hash values of log lines in Bloom filter structures during the data ingestion phase. This preliminary action transforms the data into a query-optimized format that enables fast O(1) lookups without requiring decompression or complex processing during query operations.
Data Source
AI summary
A computer-implemented method is presented for storing log data generated in a distributed computing environment. The method includes: receiving a data element from a log line, where the data element is associated with a given storage entity containing the log line; applying a hash function to the data element to generate a hash value; updating a listing of storage entities with the given entity, where entries in the listing of entities are configured to identify more than one entity and each entry in the listing of entities specifies a unique set of entities; and storing the hash value, along with an address, in a probabilistic data structure, where the address maps the hash value to an entry in the listing of entities.