Column-Wise Table Compression for Faster Random Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database technologies face challenges in efficiently accessing and storing large datasets due to high storage costs and slow access times, particularly in exceptionally large databases where data compression methods do not adequately address the need for faster access without increasing expenses.
Innovation Solution
The approach involves partitioning a database table into blocks, creating a block index, and compressing these blocks using either row-wise or column-wise methods, with the option to transpose blocks for enhanced compression efficiency, allowing for faster storage and retrieval by utilizing appropriate compression algorithms for each column or row.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in exceptionally large databases using traditional methods, then storage capacity is sufficient, but storage cost increases and access speed decreases
Solution Approach 1:
The patent divides large database tables into smaller blocks and further segments them into row-wise and column-wise transposed versions. This segmentation allows selective compression of different data portions using appropriate algorithms, reducing overall storage requirements while maintaining fast access capability through strategic placement of frequently accessed blocks in faster memory.
Solution Approach 2:
The patent applies different compression algorithms and parameters to different data blocks based on their characteristics. By transposing blocks and applying compression with varying parameters (row-wise vs column-wise), the system achieves higher compression ratios for the same data, reducing storage cost while enabling faster access through reduced data transfer requirements.
2Speed
If faster memory types such as solid-state memory are used for storage, then data access speed improves, but storage cost increases
Solution Approach 1:
The patent segments database tables into blocks that can be selectively compressed and stored in different memory types. By compressing blocks efficiently, the system can store more data in faster memory types without proportionally increasing cost, as the compressed data occupies less physical space.
Solution Approach 2:
The patent creates a nested structure where compressed column-wise transposed blocks are stored within the database storage hierarchy. This nesting allows multiple levels of compression and storage optimization, enabling the system to use faster memory types for critical data blocks while maintaining cost efficiency overall.
3Quantity of substance
If data compression is applied to reduce storage requirements, then storage cost decreases, but access time increases due to decompression overhead
Solution Approach 1:
The patent divides data into blocks that are compressed independently. This segmentation allows the system to decompress only the specific blocks needed for a given query rather than decompressing entire datasets, significantly reducing access time while maintaining high compression ratios for stored data.
Solution Approach 2:
The patent pre-compresses data blocks and stores them in an organized hierarchy with block indices. This preliminary compression and organization eliminates the need for decompression during normal access operations, as the compressed blocks can be directly accessed and processed in compressed format where applicable.
4Quantity of substance
If traditional compression algorithms are used on entire tables, then storage space is reduced, but access efficiency decreases due to lack of selective access
Solution Approach 1:
The patent segments tables into blocks with block-level compression, enabling selective access to individual blocks. This segmentation maintains storage space efficiency through compression while dramatically improving access efficiency by allowing the system to retrieve only the specific blocks containing needed data rather than decompressing and scanning entire tables.
Solution Approach 2:
The patent applies different compression strategies and algorithms to different blocks based on their local characteristics (row-wise vs column-wise data patterns). This local optimization maintains high compression ratios for each block while enabling efficient selective access to blocks with appropriate compression schemes for the type of data they contain.
Data Source
AI summary
A table, such as a database table can be partitioned into blocks that are conveniently sized for storage and retrieval. The amount of storage space required and the speed of storing and retrieving blocks is proportional to the size of the blocks. Compressing the blocks leads to less required space and more speed. The columns in a table, and therefore the rows in a transposed block, tend to contain similar data. Compression algorithms can work more efficiently when sequential data items are similar. Therefore, transposing the blocks before compression or compressing them in a column-wise manner leads to better compression. Different compression algorithms can be used for each set of columnar data to yield even better compression.