Matrix Differential Compression for High-Dimensional Sensing Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods, such as PAQ8, struggle to efficiently compress high-dimensional data like optical fiber sensing data due to high calculation complexity and insufficient compression ratios, especially when dealing with large amounts of data.
Innovation Solution
A data processing method involving differential operations on rows and columns of a matrix, followed by entropy encoding, to reduce correlation and enhance compression efficiency, utilizing correlation coefficients to determine sign bits for differential operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional data compression methods (e.g., PAQ8) are used on high-dimensional data, then compression is performed, but calculation complexity is high and compression ratio is insufficient
Solution Approach 1:
The patent segments high-dimensional data into multiple low-dimensional subspaces through orthogonal transformation (e.g., discrete cosine transformation). Each subspace is processed independently with simpler compression algorithms, avoiding the high computational complexity of directly compressing the entire high-dimensional data while maintaining effective compression ratios.
Solution Approach 2:
The patent transforms data from high-dimensional space to low-dimensional subspaces through orthogonal transformation. This dimensionality reduction allows standard compression algorithms to work more efficiently on each subspace, reducing overall calculation complexity while preserving data characteristics for effective compression.
2Productivity
If differential operations are performed multiple times on data, then correlation is reduced and compression efficiency improves, but processing time increases
Solution Approach 1:
The patent performs orthogonal transformation (preliminary action) to convert data into low-dimensional subspaces before applying differential operations. This preliminary transformation reduces the data dimensionality and complexity, allowing differential operations to achieve the desired correlation reduction with fewer iterations and less processing time.
Solution Approach 2:
By transforming data into low-dimensional subspaces first, the patent enables differential operations to work more efficiently. The reduced dimensionality means fewer operations are needed to achieve effective correlation reduction, balancing compression efficiency improvement with acceptable processing time.
Data Source
AI summary
This application provides a data processing method and apparatus, and relates to the field of data compression. The method includes: obtaining to-be-compressed data; sequentially performing n times of preset processing on the to-be-compressed data to obtain preprocessed data, where the preset processing includes: performing a differential operation on rows of a to-be-operated matrix, or performing a differential operation on columns of the to-be-operated matrix, where the to-be-operated matrix is a matrix obtained through a previous time of the preset processing, or the to-be-operated matrix is a matrix formed by the to-be-compressed data; and compressing the preprocessed data through entropy encoding, to obtain compressed data.