Adaptive Data Compression for Resource-Aware Big Data Migration

Solution Overview

Problem

Current big data migration in cloud computing is inefficient due to high compression and decompression times, which are not optimized by existing methods that fail to consider real-time system resources and information entropy, leading to suboptimal use of CPU, memory, and network bandwidth.

Innovation Solution

A method that identifies the information entropy of data blocks and determines a preferred compression method based on real-time resource status, including CPU and memory usage of source and target systems and network bandwidth, to minimize the total time of compression, transmission, and decompression by adaptively selecting suitable compression algorithms for each data block.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If traditional compression methods are used for big data migration, then data transmission size is reduced, but compression time and decompression time increase significantly

Engineering Contradiction:

Improvedata transmission sizeVSAvoidcompression time and decompression time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent applies dynamics by making the compression method selection adaptive and dynamic rather than static. The system dynamically selects compression algorithms based on real-time resource status (CPU usage, memory availability, network bandwidth) and data characteristics (information entropy), allowing the compression strategy to adapt to changing conditions during data migration, thereby optimizing the balance between compression time and transmission size reduction

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent changes parameters by using information entropy as a key parameter to characterize data compressibility and by adjusting compression algorithm selection based on resource status parameters. The system calculates information entropy for different data blocks and uses this parameter alongside resource metrics to determine optimal compression settings, transforming the compression process from a fixed-parameter operation to a parameter-driven adaptive process

Inventive Principle:

Principle #35Parameter changes

2Speed

If high compression ratio algorithms are used, then data transmission time is reduced, but CPU and memory resource consumption increases

Engineering Contradiction:

Improvedata transmission speedVSAvoidCPU and memory resource consumption

Core Design Contradiction:

SpeedVSUse of energy by moving object

Solution Approach 1:

The patent implements feedback by continuously monitoring resource status (CPU usage, memory availability, network bandwidth) during the data migration process and using this feedback to adjust compression algorithm selection. The system creates a closed-loop control mechanism where resource consumption metrics feed back into the decision-making process, allowing the system to respond to resource constraints and prevent excessive CPU and memory usage while maintaining efficient transmission speeds

Inventive Principle:

Principle #23Feedback

3Productivity

If compression optimization is performed without considering real-time resource status, then compression speed may improve, but overall system performance and resource utilization become unbalanced

Engineering Contradiction:

Improvecompression speedVSAvoidresource utilization balance

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent applies dynamics by making the compression method selection adaptive and dynamic rather than static. The system dynamically selects compression algorithms based on real-time resource status (CPU usage, memory availability, network bandwidth) and data characteristics (information entropy), allowing the compression strategy to adapt to changing conditions during data migration, thereby optimizing the balance between compression time and transmission size reduction

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent changes parameters by using information entropy as a key parameter to characterize data compressibility and by adjusting compression algorithm selection based on resource status parameters. The system calculates information entropy for different data blocks and uses this parameter alongside resource metrics to determine optimal compression settings, transforming the compression process from a fixed-parameter operation to a parameter-driven adaptive process

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS9521218B1Adaptive compression and transmission for big data migration

Publication Date: 2016.12.13 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method for optimizing migration efficiency of a data file over network is provided. Specifically, a total time of compression time of the data file, transfer time of the data file over the network, and decompression time of the data file, is minimized by adaptively selecting compression methods to compress each data block of the data file. For selecting a compression method for a data block, information entropy of the data block is analyzed, and a real status of computing and system resources is considered. Further, trade-off among the resource usage, compassion speed and compression ratio is made to calculate an optimized transmission solution over the network for each data block of the data file.