Compiled Table Algorithm for Database Column Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Horizontal database storage methods are inefficient when only a subset of columns is retrieved, leading to reduced data density and increased I/O costs due to unnecessary data being read from disk, and existing compression algorithms are not effective for database rows as they fail to exploit column correlations.
Innovation Solution
The Compiled Table Algorithm (CTA) treats compression as a compilation process, generating instructions based on column values from adjacent rows, allowing each column to be compressed independently, reducing storage needs and improving performance by only decompressing relevant columns during scans.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If horizontal storage is used with row-oriented data, then data loading and SQL create table actions are straightforward, but data density is reduced and I/O costs increase when only a subset of columns is retrieved
Solution Approach 1:
The patent divides the horizontal row structure into vertical column segments. Instead of storing complete rows horizontally, each column is stored as a separate vertical segment. This allows selective retrieval of only the needed columns, improving data density and reducing I/O costs while maintaining ease of data loading through the column-oriented segmentation approach.
2Quantity of substance
If classic compression algorithms are applied to database rows, then storage compression is achieved, but compression effectiveness is reduced due to failure to exploit column correlations
Solution Approach 1:
The patent segments the compression process by column, applying compression algorithms to each vertical column segment independently. This segmentation allows the compressor to exploit correlations within each column while maintaining the horizontal row structure for retrieval. The column-oriented segmentation enables effective compression without sacrificing compression effectiveness.
Solution Approach 2:
The patent applies local quality by treating each column differently based on its specific correlation characteristics. Instead of uniform compression, the system adapts compression parameters and methods to the local properties of each column, maximizing compression effectiveness for each column while maintaining overall storage efficiency.
3Reliability
If all columns are stored horizontally, then complete row data is available, but unnecessary data must be read from disk increasing I/O costs
Solution Approach 1:
The patent segments the storage structure into vertical column segments that can be independently retrieved. This segmentation enables the system to read only the necessary columns from disk, reducing I/O costs and energy consumption while maintaining data availability through the segmented column structure that can be selectively accessed.
Solution Approach 2:
The patent extracts only the needed columns from the full row structure during retrieval operations. By separating columns from the horizontal row structure, the system can extract and read only the necessary data, eliminating the need to read unnecessary data from disk and thereby reducing I/O costs while maintaining reliable access to required data.
4Manufacturing precision
If vertical compression is implemented, then compression effectiveness is improved, but device complexity increases
Solution Approach 1:
The patent segments the compression implementation into modular column-oriented units that can be processed independently. This segmentation reduces overall system complexity by allowing parallel processing and independent optimization of each column's compression, while maintaining high compression effectiveness through the segmented vertical structure.
Solution Approach 2:
The patent implements a universal compression framework that can handle multiple data types and column structures through a single multi-functional system. This universality reduces complexity by providing a unified approach to compression that works across different column types, eliminating the need for separate compression implementations for each data type while maintaining compression effectiveness.
Data Source
AI summary
An approach for providing compression of a database table that uses a compiled table algorithm (CTA) that provides leverage. Data within any given column in adjacent rows is often the same as or closely related to its neighbors. Rather than storing data in each column of each row as a specific integer, floating point, or character data value, a field reconstruction instruction is stored that when executed by a decompression engine can reconstruct the data value. The field reconstruction instruction may be bit granular and may depend upon past history given that the data compression engine may preserve state as row data is streamed off a storage device.