Adaptive Compression Filters for Patterned Numeric Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression techniques are suboptimal for compressing patterned numerical data, particularly in the 'big data' era, leading to inefficient storage and retrieval costs and query execution performance.
Innovation Solution
The technology employs a dynamic sequence of compression filters that identify and encode common patterns and patterns of patterns in numeric data, using a combination of manually chosen and dynamically selected filters to achieve high compression ratios, including new algorithms like Dictionary Run-Length Encoding and Week Of Encoding, to optimize data compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data compression techniques are used, then storage space is reduced, but compression ratio is insufficient for patterned numerical data
Solution Approach 1:
The patent applies dynamics by transitioning from static compression filters to a dynamic sequence where filters are selected and ordered based on data characteristics. The system dynamically determines the optimal compression approach by analyzing data patterns and adapting the filter sequence accordingly, enabling high compression ratios for patterned numerical data while maintaining effectiveness across different data types.
Solution Approach 2:
The patent changes parameters by introducing multiple compression filters with different parameters (e.g., dictionary size, run-length thresholds, pattern matching sensitivity) and selecting appropriate parameter sets based on data characteristics. This allows the system to optimize compression ratios for specific patterns in numerical data while adapting to various data types and compression requirements.
2Ease of manufacture
If static compression filter configuration is used, then implementation is simple, but compression effectiveness varies suboptimally across different data types
Solution Approach 1:
The patent achieves universality by designing a compression system that can handle multiple data types and patterns through a single unified architecture. The sequence of compression filters includes diverse algorithms (e.g., run-length encoding, dictionary-based compression, pattern matching) that can adaptively process different data types, making the system universally applicable while maintaining implementation simplicity through standardized filter interfaces.
Solution Approach 2:
The patent applies segmentation by dividing the compression process into multiple independent filter stages, each specialized for detecting specific patterns in numerical data. This segmentation allows the system to tackle different data patterns with appropriate specialized filters while maintaining overall implementation simplicity through modular design and standardized filter interfaces.
3Duration of action of stationary object
If more data is stored on cloud, then data retention period increases, but storage and retrieval costs increase proportionally
Solution Approach 1:
The patent changes the parameter of data representation by applying compression algorithms that transform numerical data into more compact forms. By detecting patterns and encoding them efficiently, the system reduces the storage parameter while maintaining data integrity and retrievability, thereby extending data retention periods without proportional increases in storage costs.
4Quantity of substance
If data compression is applied, then storage cost is reduced, but query execution performance may be affected
Solution Approach 1:
The patent applies preliminary action by pre-processing data through compression during data ingestion or batch operations, rather than compressing on-demand during queries. The system also pre-computes and stores metadata about data patterns and compression characteristics, enabling quick determination of optimal decompression strategies during query execution, thereby minimizing query execution time while maintaining storage cost reductions.
Data Source
AI summary
Certain implementations of the disclosed technology may include methods and computing systems for performing high-density data compression, particularly on numerical data that demonstrates various patterns, and patterns of patters. According to an example implementation, a method is provided. The method may include extracting a data sample from a data set, compressing the data sample using a first compression filter configuration, and calculating a compression ratio associated with the first compression filter configuration. The method may also include compressing the data sample using a second compression filter configuration and calculating a compression ratio associated with the second compression filter configuration. A particular compression filter configuration to utilize in compressing the entire data set may be selected based on a comparison of the compression ratio associated with the first compression filter configuration and a compression ratio associated with the second compression filter configuration.