SuperMinHash Log Signature Calculation with Hybrid Random Numbers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current automated log data analysis systems face challenges in efficiently calculating locality-sensitive signatures due to high time and space complexity, which hinders the accurate determination of similarity between log entries, especially for complex data sets.
Innovation Solution
The proposed method uses a combination of floating-point and integer pseudo-random numbers to calculate SuperMinHash signatures, optimizing the runtime complexity and accuracy by interleaving random number generation and permutation calculation, and maintaining a histogram to terminate the calculation early when maximum signature values are reached.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If MinHash algorithm is used to calculate locality-sensitive signatures, then the similarity between log entries can be determined, but the time complexity and space complexity increase significantly
Solution Approach 1:
The patent changes the parameter representation from standard floating-point numbers to a hybrid format combining integer and floating-point components. Specifically, it uses integer pseudo-random numbers for permutation calculation and floating-point pseudo-random numbers for signature value storage, optimizing the balance between precision and computational efficiency
Solution Approach 2:
The patent segments the signature calculation process into distinct phases: integer pseudo-random number generation for permutation, floating-point pseudo-random number generation for signature values, and early termination based on histogram analysis. This segmentation allows each phase to be optimized independently
2Reliability
If more log data is collected to improve analysis accuracy, then the coverage of operating conditions increases, but the difficulty of manual analysis becomes impossible
Solution Approach 1:
The patent creates compact copies of log data in the form of locality-sensitive signatures. Instead of analyzing raw log entries directly, it generates condensed signature representations that preserve similarity information while reducing data volume, enabling automated comparison and analysis
Solution Approach 2:
The patent replaces manual mechanical analysis with automated computational systems. The signature-based approach enables machine-to-machine comparison of log patterns, substituting human analysis with algorithmic processing that can handle large volumes of data efficiently
3Reliability
If signature calculation is performed for all input elements to ensure complete coverage, then the analysis comprehensiveness improves, but the computational overhead increases
Solution Approach 1:
The patent implements early termination of signature calculation based on histogram analysis. When the histogram shows that maximum signature values have been reached, further calculation is terminated, performing only the necessary portion of computation while maintaining result accuracy
Solution Approach 2:
The patent uses histogram feedback to monitor the distribution of signature values during calculation. This feedback mechanism allows the system to detect when sufficient coverage has been achieved and terminate computation early, preventing unnecessary computational overhead
Data Source
AI summary
A system and method for the analysis of log data is presented. The system uses SuperMinHash based locality sensitive hash signatures to describe the similarity between log lines. Signatures are created for incoming log lines and stored in signature indexes. Later similarity queries use those indexes to improve the query performance. The SuperMinHash algorithm uses a two staged approach to determine signature values, one stage uses a first random number to calculate the index of the signature value that is to update. The two staged approach improves the accuracy of the produced similarity estimation data for small sized signatures. The two staged approach may further be used to produce random numbers that are related, e.g. each created random number may be larger than its predecessors. This relation is used to optimize the algorithm by determining and terminating when further created random numbers have no influence on the created signature.