Hardware Data Compression with Failover for Streaming Databases

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data compression methods for streaming database systems often compromise between reducing storage requirements and maintaining high performance, with hardware acceleration being necessary but requiring fail-safe mechanisms to ensure data integrity and random access capabilities.

Innovation Solution

A system and method utilizing compression hardware to generate tabular slices from financial data streams, compressing them into blocks, and storing both compressed and uncompressed data efficiently, with mechanisms for failover in case of hardware malfunction to ensure uninterrupted access.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If software compression solutions are used, then data compression is achieved, but performance requirements are not met and computational resources are excessively consumed

Engineering Contradiction:

Improvedata integrityVSAvoidcompression performance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent replaces software-based compression mechanisms with hardware-based compression engines. The hardware compressor is implemented as a dedicated computational device that performs compression operations in parallel with database operations, eliminating the performance bottleneck of software compression while maintaining data integrity through hardware-enforced compression algorithms.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent segments the database system into distinct functional components: a hardware compressor unit, a database management system, and a failover mechanism. The compression function is separated from the main database operations, allowing independent optimization of compression performance while maintaining database integrity and enabling parallel processing of compression and query operations.

Inventive Principle:

Principle #1Segmentation

2Productivity

If hardware compression is implemented, then compression performance is improved, but system reliability decreases due to hardware malfunction risks

Engineering Contradiction:

Improvecompression performanceVSAvoidsystem reliability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent implements a failover mechanism that prepares backup compression capabilities in advance. When hardware compression is enabled, a corresponding software compression engine is simultaneously maintained but inactive. Upon detection of hardware failure, the system automatically activates the software engine, providing a cushion against complete system failure and ensuring continuous operation.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Solution Approach 2:

The patent introduces a mediator layer that manages the interaction between hardware and software compression engines. This mediator monitors hardware status, manages failover transitions, and ensures data consistency between the two compression modes, thereby maintaining system reliability while allowing high-performance hardware compression to operate.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Quantity of substance

If data transformations are applied for compression, then storage requirements are reduced, but data access performance deteriorates and computational requirements increase

Engineering Contradiction:

Improvestorage requirementsVSAvoiddata access performance

Core Design Contradiction:

Quantity of substanceVSSpeed

Solution Approach 1:

The patent applies compression transformations preliminarily during data ingestion and storage operations. Data is compressed into a standardized format before being written to storage, creating a compressed data repository that maintains random access capabilities. This preliminary compression eliminates the need for repeated compression operations during query execution, thereby preserving data access performance while achieving storage reduction.

Inventive Principle:

Principle #10Preliminary action

4Productivity

If compression hardware is used, then compression efficiency is improved, but device complexity increases due to fail-safe mechanism requirements

Engineering Contradiction:

Improvecompression efficiencyVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent designs the compression system with universal components that serve multiple functions. The hardware compression engine can operate in both normal and failover modes, the software compression engine can serve as both backup and primary compressor, and the mediator layer handles both monitoring and activation functions. This multi-functionality reduces overall system complexity despite the presence of fail-safe mechanisms.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS7987161B2System and method for data compression using compression hardware

Publication Date: 2011.07.26 THOMSON REUTERS MARKETAB

AI summary

A system and method for data compression using compression hardware is disclosed. According to one embodiment, a computer-implemented method comprises receiving a data set in a data stream. The data set includes a set of data descriptor fields. Using the set of data descriptor fields included in the data set, one or more tabular slices and an index are generated. The one or more tabular slices are compressed into a compressed data block by a data compression scheme using a hardware compressor. A compression data file is generated using the compressed data block and stored in a database.