Compact Sparse Column Groups for Database Index Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database search index technologies face challenges in reducing memory footprint and improving access efficiency for large datasets, particularly in network-based publication systems like eBay and Amazon, where low-latency searching is crucial but inefficient due to the representation of sparse lists of attributes.
Innovation Solution
The use of a compact or compressed structure, such as a wavelet matrix, to represent attribute IDs, combined with directly-addressable variable-length codes (DACs) to reorder and represent data elements in a byte blob, allowing for quick determination of attribute presence and position, thereby reducing memory usage and enhancing access speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional representation of sparse lists of attributes is used in database search index, then the index can store all necessary data elements, but the memory footprint becomes large and access efficiency deteriorates
Solution Approach 1:
The patent segments the search index into multiple column groups, where each group contains only the non-null values for a specific attribute. This segmentation eliminates storage of null values and enables independent processing and caching of smaller column groups, reducing overall memory footprint while improving access efficiency through selective column retrieval.
Solution Approach 2:
The patent introduces a new dimensional organization by sorting column groups based on null density rather than traditional attribute ordering. This dimensional reorganization optimizes memory layout and access patterns, allowing the system to achieve both reduced memory footprint and improved access efficiency simultaneously.
2Ease of operation
If detailed length information is stored for each data element to enable proper interpretation, then data can be accurately read, but memory usage increases and access speed decreases
Solution Approach 1:
The patent extracts and eliminates the need for storing explicit length information for each data element. By using variable-length encoding schemes and leveraging the sorted nature of column groups, the system can determine element boundaries without additional metadata, thereby reducing memory usage while maintaining accurate data interpretation.
Solution Approach 2:
The data structure is designed to be self-describing through its organization and encoding, eliminating the need for external length information. The sorted order and encoding schemes inherently provide the information needed to interpret data correctly, allowing the structure to serve its own interpretation needs without additional overhead.
3Adaptability or versatility
If conventional index structures are used to ensure complete data representation, then all attributes can be represented, but the index size becomes large and slows down search operations
Solution Approach 1:
The patent applies local quality by organizing column groups with similar characteristics (null density, data type, access frequency) together and applying optimization techniques selectively to each group. This allows the index to maintain complete attribute representation while minimizing overall size through localized optimization strategies applied to individual column groups.
Data Source
AI summary
Methods and systems for improving memory utilization and access speed of database indexes are disclosed. A representation of attributes of items listed on a network-based publication system is identified. The representation includes a plurality of lists of the attributes as a sequence of serialized attributes, each of which stores an attribute identifier (ID) and data corresponding to the attribute identifier. A compact or compressed structure is used to represent the attribute IDs. This representation allows a reader application to quickly determine whether an attribute ID is present in the lists or not. Directly-addressable variable-length codes (DACs) are used, in an unconventional manner, to represent the data. By representing the data in this way, the index takes up less memory for certain types of data and the reader application is able to gain faster access to the index.