Hardware Data Compression With Indexed Tabular Slices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods for streaming database systems often compromise between reducing storage requirements and maintaining high performance, with hardware acceleration being necessary but prone to malfunctions, and lacking efficient mechanisms for random access to both compressed and uncompressed data.
Innovation Solution
A system and method that partitions data into tabular slices using descriptor fields, generates an index for rapid access, and uses hardware compression to create a compressed data block with a header storing compression scheme information, ensuring efficient random access and fail-safe operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If software compression solutions are used, then computational requirements are reduced, but performance requirements cannot be met for high-performance database systems
Solution Approach 1:
The patent replaces software-based compression mechanisms with hardware-based compression engines. The hardware compressor is implemented as a dedicated compression engine that can be integrated into the database system, providing hardware-accelerated compression that meets both performance and productivity requirements.
2Quantity of substance
If data transformations are applied to reduce storage requirements, then storage efficiency improves, but data access performance deteriorates
Solution Approach 1:
The patent applies compression transformations during the data loading phase rather than during query execution. By pre-compressing data into the database storage structure, the system reduces storage requirements while maintaining fast access performance, as the compressed data is stored in an optimized format that allows efficient retrieval without requiring real-time transformation.
3Productivity
If hardware compression is implemented, then compression performance improves, but system reliability deteriorates due to hardware malfunction risks
Solution Approach 1:
The patent implements a fail-safe mechanism that detects hardware compressor malfunctions and automatically switches to software-based compression. The system includes error detection logic that monitors hardware compressor operation and provides alternative compression paths, ensuring continuous operation even when hardware fails.
4Quantity of substance
If compression is integrated into existing database systems, then storage efficiency improves, but random access efficiency deteriorates
Solution Approach 1:
The patent divides compressed data into fixed-size blocks with associated metadata that enables efficient random access. Each compressed data block contains or is associated with information about the compression scheme used, allowing the system to quickly locate and decompress specific data segments without processing entire data sets, thus maintaining random access efficiency while achieving storage compression.
Data Source
AI summary
A system and method for data compression using compression hardware is disclosed. In accordance with the method, a data set in a data stream is received. The data set includes a set of data descriptor fields. The data set is portioned into one or more data subsets using the set of data descriptor fields. One or more tabular slices and an index are generated for at least one of the data subsets using the set of data descriptor fields. The one or more tabular slices are identified by the index. The one or more tabular slices are compressed into a compressed data block by a data compression scheme using a hardware compressor. A compression data file is generated in a database. The compression data file has a header that stores information about the data compression scheme. The compressed data block is stored in the compression data file.