Column-Wise Data Compression for Faster Partial Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression technologies in databases, such as those used in IoT sensing data, result in low compression rates and lengthy query times due to their generic approach, which does not account for the specific characteristics of data columns and requires decompressing all data for partial queries.
Innovation Solution
A data processing apparatus and method that selects appropriate compression algorithms based on the characteristics of individual data columns and assigns unique identification codes to data from different apparatuses, allowing for selective compression and decompression of data groups, enabling efficient compression and quick querying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If generic compression technology is used for sensing data, then data can be compressed, but the compression rate is low and query time is long
Solution Approach 1:
The patent segments the sensing data into multiple columns based on different characteristics (e.g., time-series data, spatial data, attribute data). Each column is compressed independently using algorithms suited to its specific characteristics, thereby improving overall compression rate while enabling selective decompression for faster queries.
Solution Approach 2:
Different compression algorithms are applied to different columns based on their local characteristics. For example, time-series columns may use delta encoding, while categorical columns use run-length encoding. This localized approach optimizes compression for each data type without compromising query performance.
2Productivity
If all sensing data is compressed using the same algorithm, then the process is simple, but the compression rate is limited
Solution Approach 1:
The system dynamically selects compression algorithms based on the characteristics of each column. The compression process adapts to the data being processed, choosing from multiple algorithms (e.g., lossless, lossy, dictionary-based) depending on the column's data type, variability, and importance, thereby achieving higher compression rates without manual intervention.
3Reliability
If all compressed data must be decompressed for partial queries, then data integrity is maintained, but query time becomes too long
Solution Approach 1:
The patent extracts and stores metadata about each column's compression characteristics and query patterns. When a query is received, the system extracts only the necessary columns for decomposition, leaving other columns compressed. This selective extraction maintains data integrity for queried portions while significantly reducing query time.
Solution Approach 2:
Instead of decompressing all compressed data, the system performs partial decomposition only on the columns required for the specific query. This partial action approach maintains data integrity for the needed information while avoiding the time cost of decompressing unnecessary data.
Data Source
AI summary
A data processing apparatus and a data processing method thereof are provided. The data processing apparatus includes a register and a processor electrically connected to the register. The register is stored with a plurality of data. The plurality of data each includes a first sub-datum and a second sub-datum. The plurality of first sub-data corresponds to a first column and the plurality of second sub-data corresponds to a second column. The processor compresses the first sub-data by a first compression algorithm according to a first characteristic of the plurality of first sub-data and compresses the second sub-data by a second compression algorithm according to a second characteristic of the plurality of second sub-data.