Tabular Data Model Size Reducer for Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Tabular data models in analytics systems often require significant storage space, limiting their ability to handle large volumes of data efficiently, as they directly reflect the data source without optimization.
Innovation Solution
A size reducer evaluates columns of the tabular data model, determining modifications such as splitting, removing information, or eliminating columns to reduce memory burden, allowing for a more compact representation that maintains effective data analysis capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the tabular data model directly reflects the data source without optimization, then the data completeness and accuracy are maintained, but the storage space requirement increases significantly
Solution Approach 1:
The patent segments the tabular data model into multiple compressed representations, including dictionary-encoded columns and run-length encoded sequences. By dividing the data storage into logical segments that can be independently compressed, the system maintains data completeness while reducing overall storage requirements.
Solution Approach 2:
The patent changes the parameter representation by transforming raw data values into compressed forms such as dictionary indices, run-length pairs, and differential encodings. This parameter transformation allows the same data information to be stored more efficiently, reducing storage space while preserving data accuracy for analysis.
2Productivity
If the tabular data model stores large volumes of data in memory, then the data availability for analysis is improved, but the memory burden increases
Solution Approach 1:
The patent implements nested compression structures where compressed data representations are embedded within the tabular model framework. Dictionary encodings, run-length sequences, and other compressed forms are nested within the overall data model structure, allowing efficient in-memory storage that maintains fast access capabilities while reducing the total memory footprint.
3Quantity of substance
If compression techniques are applied to reduce data size, then the storage efficiency is improved, but the data processing complexity increases
Solution Approach 1:
The patent applies compression techniques during the data loading and model creation phase rather than during query execution. By performing the compression transformation in advance when data is first imported into the tabular model, the system reduces data size efficiently without adding complexity to subsequent analysis operations. The compressed representation becomes the new normal form that queries operate upon directly.
Data Source
AI summary
A size reducer for tabular data models. As the tabular data model is being created, the size reducer evaluates one or more columns of the tabular data model. For a given column, the memory burden and data type of the column are determined. Based on this information, the size reducer automatically determines at least one modification that can be made to the column (as compared to the source column at the data source) in order to reduce the size of the column's burden in the tabular data model. Example modifications might include splitting of column as compared to its source column in the data source, removing information (e.g., rounding) from a column as compared to its source column, and even eliminating columns from the tabular data model that are present in the external data source.