Memory Device Vector Compression for High-Dimensional Data Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current similarity analysis methods in fields like text mining, data mining, and biometric recognition face challenges in achieving simple calculations with high analysis confidence, particularly due to high-dimensional data which requires significant computational resources and storage.
Innovation Solution
A memory device and data search method that compresses search data and database objects using vector compression techniques, such as data-equal-quantity or dimension-equal-quantity compression, to reduce dimensionality and facilitate fast matching using Hamming distance calculations, while maintaining accuracy through feature vector generation and masking low-confidence vectors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-dimensional vector data is used for similarity analysis, then analysis confidence and accuracy are improved, but computational resources and storage requirements increase significantly
Solution Approach 1:
The patent extracts only the most significant features from high-dimensional vectors by identifying and retaining vectors with the largest absolute values (top-k selection). This extraction process removes redundant information while preserving the essential characteristics needed for similarity analysis, thereby reducing computational load while maintaining analysis confidence.
Solution Approach 2:
The patent transforms the parameter representation by converting continuous high-dimensional vectors into discrete compressed formats using quantization and encoding schemes. This parameter transformation reduces the storage dimensionality and computational complexity while preserving the semantic meaning and similarity relationships in the data.
2Loss of information
If high-dimensional vector data is stored, then data completeness is improved, but storage requirements increase significantly
Solution Approach 1:
The patent extracts only the essential components of the vector data by selecting top-k significant vectors and discarding less important dimensions. This selective extraction maintains the core information needed for accurate similarity analysis while dramatically reducing the storage footprint of the database.
Solution Approach 2:
The patent applies quantization to transform continuous vector values into discrete representations with fewer bits, and uses encoding schemes to compress the stored data. These parameter changes enable efficient storage while preserving the essential information content and similarity relationships.
3Measurement precision
If traditional vector analysis methods are used, then analysis accuracy is maintained, but calculation complexity increases
Solution Approach 1:
The patent simplifies the calculation process by extracting only the top-k most significant vectors for comparison, rather than processing all high-dimensional elements. This extraction approach reduces the computational complexity of similarity calculations while maintaining analysis accuracy by focusing on the most discriminative features.
Solution Approach 2:
The patent transforms the calculation methodology by using compressed and quantized vector representations instead of full-precision floating-point arithmetic. This parameter change enables faster computations with reduced complexity while preserving the essential similarity relationships through carefully designed distance metrics adapted for compressed data.
Data Source
AI summary
A data search method for a memory device is provided. The data search method includes: based on a recorded compression mode, vectoring a search data to generate a search data vector, and based on the recorded compression mode, compressing the search data and a plurality of objects in a database; setting a search condition; searching the objects of the database by the search data vector to determine whether the search data is matched with the objects of the database; and recording and outputting at least one matched object of the database, the at least one matched object matched with the search data.