Probabilistic Data Structure for Log Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database management systems face inefficiencies in storing and querying large volumes of log data, particularly in distributed computing environments, due to the high CPU requirements and storage costs associated with indexing, which becomes unsustainable as data volumes grow.
Innovation Solution
A computer-implemented method that tokenizes log data using multiple rules to create base, combination, and n-gram tokens, applies hash functions to generate unique identifiers, and stores these in a probabilistic data structure, allowing for efficient deduplication and querying by updating a token map table and entities list.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional database management systems are used to store and index log data, then query performance is improved, but storage costs and CPU requirements increase significantly
Solution Approach 1:
The patent extracts only the essential information from log lines by tokenizing and hashing, discarding redundant data. Instead of storing complete log lines with full indexing, the system stores only hash values and token mappings, dramatically reducing storage requirements while maintaining query capability.
Solution Approach 2:
The patent creates a probabilistic data structure that stores compressed representations (hash values) of log data rather than exact copies. This allows efficient querying through the hash table while using minimal storage space, accepting a low false-positive rate in exchange for significant storage savings.
2Speed
If full indexing is performed using established DBMS like Elastic/Lucene, then query speed is improved, but the index size becomes larger than the compressed data itself
Solution Approach 1:
The patent uses a probabilistic data structure with hash tables that accept a low false-positive rate, effectively using a disposable approximation approach. This allows fast queries without the need for large, precise indexes, as the system can tolerate occasional false positives in exchange for significantly reduced storage and faster lookup.
3Adaptability or versatility
If data volume increases in distributed computing environments, then more comprehensive monitoring is achieved, but the rate of data ingest exceeds computer performance gain
Solution Approach 1:
The patent segments log data into tokens and hashes, processing only the essential features rather than entire log lines. This segmentation allows the system to handle massive data volumes efficiently, as the processing complexity is proportional to the number of tokens rather than the size of complete log entries.
Data Source
Figure 1~2
Figure 3a
Figure 3b
AI summary
A computer-implemented method is presented for storing log data generated in a distributed computing environment. The method includes: receiving a log line, where the log line is associated with a given computing source producing the log line; applying a first tokenization rule to create a plurality of base tokens; applying a second tokenization rule to create a plurality of combination tokens; applying a third tokenization rule to create a plurality of n-gram tokens; combining the plurality of tokens into a set of tokens; for each token in the set of tokens, storing a given token by applying a hash function to the given token to generate a hash value, where the given token is associated with a given computing source at which the log line was produced; updating a listing of computing entities with the given computing source, where entries in the listing of computing entities can identify more than one computing sources and each entry in the listing of computing entities specifies a unique set of computing sources; and storing the hash value, along with an address, in a token map table of a probabilistic data structure, where the address maps the hash value to an entry in the listing of computing entities. (Fig. 1)