Hierarchical Compression Units for Tabular Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for storing tabular data face challenges in achieving an optimal balance between storage space and retrieval efficiency, particularly when dealing with diverse data types such as character strings and images, as current compression techniques often yield sub-optimal results due to inconsistent compressibility across different data types.
Innovation Solution
The implementation of recursive compression units that allow for flexible storage and retrieval of tabular data, using metadata to manage compression techniques and formats at multiple levels, enabling efficient storage and manipulation of data by separating and compressing different data types independently within a hierarchical structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If tabular data is stored compressed to reduce storage space, then storage space consumption is reduced, but retrieval speed deteriorates due to decompression overhead
Solution Approach 1:
The patent divides tabular data into multiple compression units, each handling a specific data type (e.g., character strings, images, binary data). This segmentation allows selective compression where applicable while keeping other data types uncompressed, thereby reducing overall storage space without requiring full decompression of the entire dataset during retrieval operations.
Solution Approach 2:
Different compression techniques are applied to different data types based on their compressibility characteristics. Character strings receive compression treatment while images and binary data remain uncompressed. This local quality approach optimizes storage efficiency for compressible data types without incurring decompression overhead for non-compressible types, thus improving retrieval speed for frequently accessed data.
2Device complexity
If a single compression technique is applied to all data types in tabular data, then device complexity is reduced, but compression efficiency deteriorates due to inconsistent compressibility across different data types
Solution Approach 1:
The patent segments tabular data into distinct compression units based on data type categories. Each compression unit is associated with specific data types that share similar compressibility characteristics. This segmentation enables the system to apply appropriate compression techniques to each unit, achieving high compression efficiency without requiring complex centralized compression management.
Solution Approach 2:
The patent changes the parameter of compression application from a uniform global setting to a localized per-unit setting. By associating compression techniques with specific compression units based on data type parameters, the system achieves optimal compression efficiency for each data type while maintaining manageable complexity through the modular compression unit structure.
3Speed
If tabular data is stored uncompressed to maintain fast retrieval speed, then retrieval speed is improved, but storage space consumption increases
Solution Approach 1:
The patent applies compression selectively to specific compression units based on their data type characteristics. Compressible data types such as character strings are compressed to reduce storage space, while non-compressible data types such as images and binary data are kept uncompressed to maintain fast retrieval speed. This local quality approach optimizes the storage-speed trade-off at the data type level rather than applying a uniform strategy to the entire dataset.
Data Source
AI summary
A highly flexible and extensible structure is provided for physically storing tabular data. The structure, referred to as a compression unit, may be used to store tabular data that logically resides in any type of table-like structure. According to one embodiment, compression units are recursive. Thus, a compression unit may have a “parent” compression unit to which it belongs, and may have one or more “child” compression units that belong to it. In one embodiment, compression units include metadata that indicates how the tabular data is stored within them. The metadata for a compression unit may indicate, for example, whether the data is stored in row-major or column major-format, the order of the columns within the compression unit (which may differ from the logical order of the columns dictated by the definition of their logical container), a compression technique for the compression unit, the child compression units (if any), etc.