Row-Oriented Data Storage Compression via Schema-Based Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional row-oriented data storage systems lack efficient data compression techniques due to their storage format, which restricts the application of encoding methods typically used in column-oriented storage.
Innovation Solution
A data management system that optimizes storage by processing data in blocks, implementing key/value pairs, determining schemas, grouping rows with the same schema, and applying encoding techniques to values based on their data types, thereby converting and storing data in a column-oriented format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If row-oriented storage format is used, then data access for computing purposes is simplified, but data compression efficiency deteriorates
Solution Approach 1:
The patent segments the storage system into multiple formats: row-oriented storage for computing access and column-oriented storage for compression. Each segment serves its specific purpose, allowing the system to maintain both ease of data access and compression efficiency by dividing the storage function into separate components rather than using a single unified format.
Solution Approach 2:
The patent introduces an intermediary conversion layer that translates between row-oriented and column-oriented formats. This intermediary mechanism allows the system to leverage column-oriented compression techniques while maintaining row-oriented access simplicity, resolving the contradiction by adding a mediating layer that bridges the two storage approaches.
2Loss of substance
If column-oriented storage format is used, then data compression efficiency is improved, but data access for computing purposes becomes more complex
Solution Approach 1:
The system segments storage operations into separate read paths: one for row-oriented access and another for column-oriented compression. This segmentation allows each path to be optimized independently, maintaining compression efficiency while preserving access simplicity through dedicated access mechanisms.
Solution Approach 2:
An intermediary translation layer is introduced that converts column-oriented stored data into row-oriented access formats when needed. This intermediary mechanism enables the system to benefit from column-oriented compression while providing row-oriented data access simplicity, resolving the contradiction through format translation.
3Device complexity
If conventional row-oriented storage is used, then storage structure is simple, but advanced encoding techniques cannot be applied
Solution Approach 1:
The patent implements a universal storage system that supports multiple storage formats (row-oriented and column-oriented) within the same infrastructure. This multi-functionality allows the system to apply advanced encoding techniques through column-oriented storage while maintaining the simplicity of row-oriented access, thereby enhancing adaptability without sacrificing structural simplicity.
Solution Approach 2:
An intermediary format conversion mechanism is introduced that enables advanced encoding techniques to be applied to data while maintaining compatibility with conventional row-oriented storage structures. This intermediary layer allows the system to leverage sophisticated encoding methods without requiring complete structural changes, thus improving adaptability while preserving simplicity.
Data Source
AI summary
Embodiments of the present disclosure include techniques for optimizing storage of data in row-oriented data storages. A block of data comprising a plurality of rows of data. Each row of data comprises a set of values for a set of attributes. A set of unique schemas is determined based on the plurality of rows of data. A set of groups of rows of data is determined based on the set of unique schemas. For each group of rows of data in the set of groups of rows of data, the group of rows of data is stored in a column-oriented format. For each group of rows of data in the set of groups of rows of data, a set of encoding techniques are applied to values in the group of rows of data based on the sets of attributes of the group of rows of data.