Sparse Column-Aware Numeric Encoding for Reduced Data Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data encoding schemes are inefficient for column-wise data access in memory architectures, leading to high storage overhead and suboptimal performance in searching large databases.
Innovation Solution
Implement sparse column-aware encodings for numeric data types, utilizing stochastic associative memories (SAM) to enable efficient column-wise data access and perform similarity searches, leveraging m-of-n constant weight codes for integer and floating-point data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional row-based data encoding schemes are used, then memory access is simplified for row-wise operations, but storage overhead increases and column-wise access performance deteriorates
Solution Approach 1:
The patent segments data encoding into row-based and column-based representations. Each data element is encoded with both a row code and a column code, allowing the system to access data efficiently in either direction. This segmentation resolves the contradiction by providing specialized encodings for each access pattern rather than using a single universal encoding scheme.
Solution Approach 2:
The patent introduces a dual-dimension encoding approach where data is represented in both row and column dimensions simultaneously. By adding the column dimension to the traditional row-based encoding, the system enables efficient column-wise access without sacrificing row-wise access capability, thereby reducing storage overhead while maintaining ease of operation for both access patterns.
2Speed
If traditional row-based data encoding schemes are used, then row access is efficient, but column-wise search performance deteriorates
Solution Approach 1:
The patent segments the encoding system into row-specific and column-specific components. Row codes are optimized for fast row access while column codes are optimized for efficient column-wise searching. This segmentation allows each encoding type to be specialized for its intended access pattern, resolving the contradiction between row access speed and column search efficiency.
Solution Approach 2:
The patent creates a universal encoding framework that handles both row access and column search operations through a single dual-encoded data structure. The same encoded data can be efficiently accessed in either row or column direction, making the system multi-functional and eliminating the need for separate optimization strategies for different access patterns.
3Measurement precision
If dense encoding schemes are used, then all data symbols are represented with full precision, but storage overhead and processing time increase
Solution Approach 1:
The patent applies local quality by using different encoding densities for different parts of the data structure. Rather than uniformly encoding all data symbols with maximum precision, the system uses sparse encoding for column codes where precision can be reduced, and denser encoding for row codes where full precision is needed. This local differentiation maintains necessary precision while reducing overall storage overhead.
Solution Approach 2:
The patent changes the encoding parameters from fixed dense encoding to variable sparse encoding. By adjusting the encoding density parameter based on the specific data characteristics and access patterns, the system can optimize the balance between precision and storage overhead. The sparse column codes use fewer bits per symbol while maintaining sufficient precision for the intended operations.
Data Source
AI summary
Methods and apparatus for sparse column-aware encodings for numeric data types, including integer data and floating-point data (float, double, etc.). The encoding schemes are tailored to take advantage of column addressable memories such as stochastic associative memories (SAM) to enable Stochastic Associative Search (SAS), which is a highly efficient and fast way of searching through a very large database of records (order of Billions) and finding similar records to a given query record (search key). Techniques are also disclosed for performing range searches for both integer and floating-point data types. The integer or float data is converted to Hexadecimal form and encoded using an m-of-n constant weight encoding. Only the columns with set bits in search keys need to be read, which significantly reduces the number of reads required for searches.