Column-Oriented In-Memory Page Caching for Tabular Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional tabular page caching methods require data type conversion of all columns in a two-dimensional array, leading to performance degradation, especially with large datasets, and reordering columns impacts performance significantly.
Innovation Solution
Implementing column-oriented in-memory page caching using one-dimensional arrays that store only a portion of the rows for each column, eliminating the need for data type conversion and allowing columns to be reordered without significant performance impact.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional tabular page caching uses two-dimensional arrays to store data, then data can be stored in a structured format, but data type conversion is required for all columns which significantly impacts performance
Solution Approach 1:
The patent divides the traditional two-dimensional array structure into separate one-dimensional arrays for each column. Each column is stored as an independent one-dimensional array, allowing each column to maintain its native data type without conversion. This segmentation eliminates the need for uniform data type conversion while enabling efficient caching of tabular data with mixed data types.
2Adaptability or versatility
If traditional tabular page caching stores all data in a common data type, then storage is simplified, but columns cannot be reordered without significant performance impact
Solution Approach 1:
By segmenting the data into separate one-dimensional arrays for each column, the patent enables independent manipulation of each column. Columns can be reordered, filtered, or transformed without affecting the integrity or performance of other columns, as each column maintains its own data type and storage structure.
Solution Approach 2:
The patent implements a dynamic column ordering mechanism where columns can be reordered based on user needs or analysis requirements. The system maintains performance by allowing flexible column reconfiguration without requiring complete data reconversion, as each column retains its native data type in its dedicated array.
3Device complexity
If traditional tabular page caching converts all columns to a common data type, then storage structure is simplified, but performance degrades significantly with large datasets
Solution Approach 1:
The patent segments the storage structure into separate one-dimensional arrays for each column, eliminating the need for a unified common data type. This segmentation reduces the complexity of data type conversion operations while enabling efficient processing of large datasets by maintaining native data types for each column.
Solution Approach 2:
The patent changes the fundamental parameter of data storage from a unified two-dimensional array with common data type to multiple separate one-dimensional arrays, each with its own data type. This parameter change eliminates the performance bottleneck of data type conversion while maintaining storage efficiency for large datasets with diverse data types.
Data Source
AI summary
A one-dimensional array is allocated in an in-memory cache for each column in a set of tabular data. The data type of each one-dimensional array is set to be the same as the data type of the corresponding column in the tabular data. Once the one-dimensional arrays have been allocated in memory, a portion of the data from each column in the tabular data is stored in a corresponding one-dimensional array. The tabular data stored in the one-dimensional arrays in the cache may then be utilized to generate an on-screen display of a portion of the tabular data.