Hash Suppression for High-Discriminative Data Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing tools and techniques are inadequate for efficiently updating an incremental hash index and improving the performance and accuracy of data analysis in high-velocity data streams, particularly in suppressing low-discriminative hashes that complicate object differentiation.
Innovation Solution
A computer-implemented system and method that generates hashes for data objects, computes high-dimensional sparse vectors, determines a hash suppression threshold, and suppresses selected hashes to maintain an incremental hash index with high discriminative power, enhancing analysis accuracy and efficiency in tasks like retrieval and classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all generated hashes are retained in the hash index, then the hash index comprehensively represents all data objects, but the hash index size becomes excessively large and retrieval accuracy deteriorates due to low-discriminative hashes
Solution Approach 1:
The patent extracts and removes low-discriminative hashes from the hash index based on a suppression threshold. By identifying hashes that appear in too many objects (low discriminative power) and removing them, the system reduces hash index size while improving retrieval accuracy. This directly applies the extraction principle by separating harmful elements (low-value hashes) from the useful whole.
Solution Approach 2:
The patent changes the parameter of hash discriminative power by introducing a suppression threshold that dynamically filters hashes based on their occurrence frequency. By adjusting this threshold parameter, the system optimizes the balance between hash index size and retrieval accuracy, removing hashes below the threshold while retaining those above it.
2Quantity of substance
If the hash suppression threshold is set low, then more hashes are suppressed and hash index size is reduced, but fewer hashes remain to differentiate between objects
Solution Approach 1:
The patent employs feedback mechanisms where the suppression threshold is dynamically adjusted based on performance metrics. The system monitors retrieval accuracy and object differentiation capability, and adjusts the threshold accordingly to maintain optimal performance. This feedback loop ensures that hash suppression does not degrade the ability to differentiate between objects.
Solution Approach 2:
The suppression threshold is made dynamic rather than static, allowing it to adapt to different data distributions and query patterns. The system can adjust the threshold based on the specific characteristics of the data being processed, optimizing the balance between hash index size reduction and maintaining object differentiation capability for different scenarios.
3Productivity
If conventional hash indexing is used without suppression, then all hashes are available for analysis, but processing time increases and analytical efficiency decreases due to the volume of low-value hashes
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing the suppression threshold and identifying low-discriminative hashes before actual retrieval operations. This preliminary filtering of hashes occurs during an initialization or preprocessing phase, so that during subsequent analytical operations, the system only processes the reduced set of high-value hashes, significantly improving analytical efficiency and reducing processing time.
Data Source
AI summary
An example method is provided in according with one implementation of the present disclosure. The method comprises generating, via a processor, a set of hashes for each of a plurality of objects. The method also comprises computing, via the processor, a high-dimensional sparse vector for each object, where the vector represents the set of hashes for each object. The method further comprises computing, via the processor, a combined high-dimensional sparse vector from the high-dimensional sparse vectors for all objects and computing a hash suppression threshold. The method also comprises determining, via the processor, a group of hashes to be suppressed by using the hash suppression threshold, and suppressing, via the processor, the group of selected hashes when performing an action.