Outlier Detection in Streaming Data via Locality Sensitive Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning and statistical algorithms are inadequate for efficiently identifying anomalous data in high-volume streaming data sets, as they were primarily designed for static data or are difficult to scale for real-time analysis.
Innovation Solution
The implementation of locality sensitive hashing (LSH) based methods, including banded LSH and min-hash signatures, to efficiently detect outliers in streaming data by using probabilistic approximations of inter-record distances and reducing storage requirements through space-optimizing techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional machine learning and statistical algorithms are used for outlier detection, then detection accuracy may be maintained, but processing speed and scalability deteriorate significantly with high-volume streaming data
Solution Approach 1:
The patent replaces traditional mechanical computation methods (exact distance calculations, full data scans) with probabilistic hashing-based methods. Locality-sensitive hashing and MinHash signatures substitute for conventional statistical algorithms, enabling approximate outlier detection that scales to streaming data volumes while maintaining acceptable accuracy through probabilistic guarantees.
Solution Approach 2:
The patent changes the fundamental parameters of outlier detection by shifting from exact distance metrics to hash-based similarity measures. By transforming the problem space using hashing functions and adjusting parameters like hash table size, bucket counts, and similarity thresholds, the system achieves scalable processing speed while preserving detection capability through controlled approximation.
2Reliability
If comprehensive data analysis is performed to reduce false positives and negatives, then detection reliability improves, but computational complexity and processing time increase
Solution Approach 1:
The patent substitutes complex mechanical analysis procedures with hashing-based probabilistic methods. Instead of performing comprehensive pairwise comparisons or iterative statistical analyses, the system uses Locality-sensitive hashing and MinHash signatures to efficiently estimate similarity, reducing algorithmic complexity while maintaining reliability through probabilistic bounds on false positives and negatives.
Solution Approach 2:
The patent creates simplified copies of the original data through hash signatures and bucket representations. These compact hashed versions serve as proxies for the full data, enabling reliability checks on compressed representations that are computationally tractable, thereby reducing complexity while preserving essential detection capabilities.
3Loss of information
If streaming data is stored in full for analysis, then complete data availability is maintained, but storage costs and memory requirements increase significantly
Solution Approach 1:
The patent extracts essential similarity information from the full streaming data and stores only these extracted features in hash tables and signatures. By taking out only the necessary components (hash values, MinHash signatures, bucket assignments) rather than storing complete data records, the system maintains sufficient information for outlier detection while dramatically reducing storage volume requirements.
Solution Approach 2:
The patent creates compact hashed copies of streaming data that occupy minimal storage space. These hash-based representations (Locality-sensitive hash buckets, MinHash signatures) serve as space-efficient proxies that preserve the essential similarity structure needed for outlier detection, enabling complete analytical capability with fraction of the original storage requirements.
Data Source
AI summary
A matching record set with respect to a particular data record of a stream is identified based on output values produced by a particular band of locality sensitive hash functions. Using respective matching record sets corresponding to the particular data record and one or more other bands of locality sensitive hash functions, an estimate of a count of data records of the stream which meet a particular inter-record distance criterion is obtained. A determination as to whether the particular data record is to be designated as an outlier with respect to previously-observed records of the data stream is made using the estimated count.