Columnar Object Storage for Cache-Efficient Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing object-oriented programming systems face inefficiencies in memory allocation and access due to the allocation of storage space based on objects, leading to low reading efficiency and increased cache misses.
Innovation Solution
A data processing method that stores and reads objects by grouping attribute values into data columns, compressing redundant information, and utilizing consecutive storage spaces to improve spatial locality, reducing memory usage and cache reads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If storage space is allocated based on individual objects, then each object can be stored independently with its complete attributes, but reading efficiency decreases and cache misses increase when accessing multiple objects of the same class
Solution Approach 1:
The patent segments object attributes into separate data columns, where each column stores a specific attribute (e.g., name, age, address) for all objects of a class. This segmentation allows the system to read only the required attribute columns when accessing multiple objects, rather than loading entire object instances, thereby improving reading efficiency and reducing cache misses while maintaining the ability to access individual object attributes independently.
2Reliability
If complete object instances are stored, then all attributes are available for each object, but redundant information is stored and memory usage increases
Solution Approach 1:
The patent extracts common attributes from individual object instances and stores them in separate data columns. Instead of storing complete object instances with all attributes repeated for each object, the system extracts attributes into columns and uses row identifiers to reference specific objects. This extraction eliminates redundant storage of identical attribute values across multiple objects while ensuring all attributes remain available for access.
3Adaptability or versatility
If objects are stored in non-consecutive memory locations, then object independence is maintained, but spatial locality deteriorates and cache performance decreases
Solution Approach 1:
The patent transitions from storing objects in a traditional row-oriented manner (where each object occupies a contiguous block of memory) to a column-oriented storage structure. In this new dimensional arrangement, attributes are stored in vertical columns across memory, allowing objects to be referenced by row identifiers while their attributes are accessed through columnar data structures. This dimensional change improves spatial locality for attribute access patterns while maintaining object independence through the row identifier mechanism.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Embodiments of this application provide a data processing method and a related device. The method includes: obtaining N objects that belong to a same class; determining M groups of data based on attribute values of M attributes included in the N objects, where the M groups of data are in a one-to-one correspondence with the M attributes, and each of the M groups of data indicates attribute values of a corresponding attribute of the N objects; and storing the M groups of data by using M storage space sets respectively. In the foregoing technical solutions, attribute values of a same attribute of the N objects that belong to the same class are stored by using a same storage space set. Therefore, the foregoing technical solutions provide good data locality. In addition, in the foregoing technical solutions, only an attribute value of an object or data indicating an attribute value may be stored without storing some redundant information (for example, an object header and alignment padding). Therefore, the foregoing technical solutions save storage space.