Data Chunk Prioritization for Reducibility-Aware Network Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data reduction approaches for network transmissions are sub-optimal as they fail to consider the actual content of the data and are resource-intensive, leading to inefficient compression and de-duplication processes.
Innovation Solution
A method that prioritizes data chunks based on their reducibility score, determined by compressibility and de-duplication potential, using a priority queue to focus resource-intensive reduction operations on chunks with the highest potential, while transmitting irreducible chunks first to optimize the tradeoff between reduction ratio and resource expenditure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data chunks are compressed using multiple engines to find the best fit, then compression ratio is improved, but resource consumption and time increase substantially
Solution Approach 1:
The patent assigns a reducibility score to each data chunk before the compression process based on analysis of the data content. This preliminary assessment identifies which chunks are most suitable for compression, allowing the system to prioritize reduction operations on high-reducibility chunks and avoid wasting resources on low-reducibility chunks, thereby improving overall transfer efficiency without sacrificing compression ratios
2Quantity of substance
If compression operations are performed on all data chunks, then compression ratio is improved, but resource consumption increases
Solution Approach 1:
The patent applies different compression strategies to different data chunks based on their individual reducibility scores. High-reducibility chunks receive aggressive compression with multiple engines, while low-reducibility chunks are either compressed with a single engine or transmitted without compression. This localized, differentiated approach optimizes the balance between compression ratios and resource consumption by tailoring the compression intensity to the specific characteristics of each data chunk
3Quantity of substance
If data is cut into chunks and compressed with multiple engines, then compression ratio is improved, but the process becomes slow and resource intensive
Solution Approach 1:
The patent applies compression operations selectively rather than universally. By using reducibility scores to identify the most compressible chunks, the system applies full multi-engine compression only to those chunks with high reduction potential, while using lighter or no compression on other chunks. This partial application of compression resources achieves meaningful overall compression ratios while significantly reducing total compression time and resource expenditure
Data Source
AI summary
Methods and systems for data transfer include adding a data chunks to a priority queue in an order based on utilization priority. A reducibility score for the data chunks is determined. A data reduction operation is performed on a data chunk having a highest reducibility in the priority queue using a processor if sufficient resources are available. The data chunk having the lowest reducibility score is moved from the priority queue to a transfer queue for transmission if the transfer queue is not full.