Bit-Vector Record Compression for Parallel Processing Without Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods in database systems require decompression for processing, leading to increased processing time and memory requirements, as they can only be processed sequentially and require large random access memory for intermediate results.
Innovation Solution
Divide binary data records into bit vectors, segmenting them into equal-sized partial areas, classify and combine non-trivial and quasi-trivial partial areas into R and O blocks, respectively, allowing for parallel processing without decompression by noting absolute positions and counts within these blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is applied to reduce data size, then storage space is reduced, but processing time increases due to required decompression
Solution Approach 1:
The patent divides binary data records into multiple bit vectors, each representing a specific attribute or field. This segmentation allows independent processing of individual bit vectors without requiring complete decompression of the entire data structure, thereby reducing processing time while maintaining compression benefits.
Solution Approach 2:
The patent performs preliminary classification of partial areas within bit vectors into trivial, quasi-trivial, and non-trivial categories during the compression phase. This preliminary organization enables direct processing of compressed data structures without full decompression, as the classification metadata stored during compression facilitates efficient query execution on compressed data.
2Ease of operation
If sequential decompression is performed for processing, then data can be processed, but large random access memory is required for intermediate results
Solution Approach 1:
By segmenting data into bit vectors with classified partial areas, the patent enables processing of only relevant segments rather than loading entire decompressed datasets into memory. The classification metadata allows selective access to specific bit vectors and partial areas, dramatically reducing memory requirements.
Solution Approach 2:
The patent implements partial processing by allowing operations on compressed data structures without complete decompression. Only the necessary bit vectors and partial areas required for specific queries are processed, rather than decompressing and loading all data into memory, thus reducing memory consumption while maintaining operational capability.
3Quantity of substance
If data records are compressed without losing information, then storage efficiency improves, but the complexity of data structure increases
Solution Approach 1:
The patent organizes compressed data into structured bit vectors with clearly defined partial areas (trivial, quasi-trivial, non-trivial). This structured segmentation provides a systematic framework that manages complexity through regular patterns and metadata, making the compressed structure more manageable than unstructured compression formats.
Solution Approach 2:
The patent transforms the data representation by changing parameters such as organizing data into bit vectors with specific partial area classifications. This parameter transformation creates a standardized compressed format with inherent structure, reducing the perceived complexity through consistent organization patterns and metadata schemas.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
System and method to compress data records by providing data records with a binary structure; dividing the data records into several bit vectors; reducing the size of each bit vector by dividing the bit vector into consecutive partial areas of equal size, each partial area consisting of n bits, classifying the partial areas as trivial partial areas, quasi-trivial partial areas and non-trivial partial areas, combining one non-trivial or several consecutive non-trivial partial areas into one so named R block, and removing the trivial partial areas; as well as combining one quasi-trivial or several consecutive quasi-trivial partial areas into one so named O block.