Dynamic Persistence Format Selection for Column Loadable Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional methods for managing data persistence formats in column-based data storage lack flexibility, leading to inefficiencies in storage utilization and retrieval performance due to fixed persistence strategies that do not account for the variability of data components.
Innovation Solution
A dynamic approach is implemented to select the optimal data persistence format based on factors like column size, dataset size, and performance requirements, involving accurate size estimation, threshold definitions, and rule-based selection between serial and unified persistence formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a uniform persistence strategy is used for all columns, then implementation simplicity is maintained, but storage efficiency deteriorates due to space consumption
Solution Approach 1:
The patent segments columns into different categories (small columns, large columns, column loadable columns) and applies different persistence strategies to each segment. Small columns use serialized storage while large columns use page-based storage, optimizing storage efficiency for each category rather than using a uniform approach for all columns.
Solution Approach 2:
The patent introduces dynamic threshold parameters (minSizeForUnifiedStorage, maxSizeForSerialStorage) that allow the persistence strategy to adapt based on column size. The system dynamically selects between serialized and page-based persistence formats based on whether column sizes fall below or exceed these thresholds, rather than using a static uniform strategy.
2Loss of substance
If serialized persistence format is used for small columns, then storage space is optimized, but retrieval performance deteriorates due to loading bottlenecks
Solution Approach 1:
The patent dynamically selects persistence formats based on column size thresholds. For column loadable columns that exceed maxSizeForSerialStorage, the system automatically chooses page-based persistence format which enables faster retrieval by loading only required pages into memory, avoiding the bottleneck of loading entire serialized columns.
Solution Approach 2:
Different retrieval optimization strategies are applied to different column types. Column loadable columns use page-based storage with selective page loading, while small columns use serialized storage. This local optimization ensures each column type gets the most appropriate retrieval strategy for its characteristics.
3Speed
If page-based persistence format is used for large columns, then retrieval performance is improved, but storage overhead increases
Solution Approach 1:
The patent segments columns based on size and access patterns, applying page-based persistence only to large columns and column loadable columns that benefit from selective loading. Small columns continue to use serialized persistence, avoiding unnecessary storage overhead from page structures for data that doesn't require them.
Solution Approach 2:
The system uses configurable threshold parameters (minSizeForUnifiedStorage, maxSizeForSerialStorage) to control when page-based persistence is applied. These parameters can be adjusted to balance storage overhead versus retrieval performance based on specific system requirements and data characteristics.
4Productivity
If dynamic format selection is implemented, then storage efficiency and retrieval performance are optimized, but system complexity increases
Solution Approach 1:
The patent implements dynamic format selection through configurable threshold parameters that automatically determine the appropriate persistence format based on column size. This dynamic approach optimizes storage efficiency and retrieval performance without requiring complex manual configuration or analysis of each column.
Solution Approach 2:
The system uses adjustable parameters (minSizeForUnifiedStorage, maxSizeForSerialStorage, dictionarySizeThreshold) to control format selection behavior. These parameters provide a simple interface for optimizing performance across different scenarios without increasing fundamental system complexity, as the same decision-making logic applies regardless of data characteristics.
Data Source
AI summary
A system for managing data persistence formats in a computing system, comprising, at least one data processor; and at least one memory result in operations comprising: computing an estimated serial size (ES) for a dataset; computing an estimated unified size (EU) for the dataset; selecting, based at least in part on a predefined set of rules, between a serial persistence format and a unified persistence format to store the dataset, wherein the selection is based at least in part on the computed estimated serial size and the computed estimated unified size; storing the selected persistence format in a metadata associated with the stored dataset; and in response to a loading request, loading the stored dataset by referring to the metadata.