Auto-Configurable Data Compression for Attribute-Specific Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods are suboptimal due to their rigidity and inability to adapt to emerging data types or new algorithms, as they require pre-determined data fields and hardcoded encoding and decoding processes, leading to inefficient compression of large data files like genomic data.
Innovation Solution
A system that automatically identifies and configures data compressors based on compression performance by selecting suitable compression types and configurations for each attribute in a data file, generating a compression parameter data structure for efficient compression and decompression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If generic compressors are applied directly to data files, then the compression process is simple and fast, but the compression ratio is suboptimal because they cannot exploit the similarity or predictability of specific data fields
Solution Approach 1:
The patent segments the data file into multiple fields with distinct characteristics, allowing each field to be compressed individually with a compressor optimized for its specific data model. This segmentation enables the system to exploit the similarity or predictability of specific data fields while maintaining overall compression efficiency.
Solution Approach 2:
The patent applies different compression algorithms and parameters to different fields based on their local characteristics. Each field receives a customized compression approach tailored to its specific data model, rather than applying a uniform compression method across the entire file.
2Loss of substance
If data fields and compressors are pre-determined for each file type with hardcoded encoding and decoding processes, then compression performance is optimized for known file types, but the system becomes rigid and cannot accommodate emerging data types or new algorithms
Solution Approach 1:
The patent implements a dynamic compressor selection mechanism that automatically identifies the most suitable compressor for each field based on its data characteristics. This dynamic approach replaces static, hardcoded configurations, allowing the system to adapt to emerging data types and new compression algorithms without requiring software modifications.
Solution Approach 2:
The system performs self-configuration by automatically analyzing data fields and selecting appropriate compressors without requiring manual intervention or pre-programmed knowledge of specific file types. This self-service capability enables the system to handle both known and emerging data types effectively.
3Loss of substance
If multiple compression types are tested for each attribute to determine the most suitable one, then compression performance is optimized, but the compression process becomes more complex and time-consuming
Solution Approach 1:
The patent applies multiple compression types to each attribute but only retains the best-performing compressor for final use. The testing of multiple compressors is performed selectively and systematically, balancing the need for optimization with acceptable processing time by not exhaustively testing all possible compression variants.
Data Source
AI summary
A method (100) for compressing and decompressing a data file, comprising: (i) receiving (120) a data file for compression comprising a plurality of different attributes; (ii) identifying (130) a first attribute of the plurality of different attributes; (iii) selecting (140) a plurality of compression types and/or configurations; (iv) compressing (150) at least some of the data from the received data file for the identified first attribute using each of the selected plurality of compression types and/or configurations; (v) determining (160) which one of the selected plurality of compression types and/or configurations is most suitable for compression; (vi) generating (170) a compression parameter data structure comprising an identification of the selected plurality of compression types and/or configurations; (vii) compressing (180) the data from the received data file for the first attribute to generate a compressed data file; and (viii) storing (190) the compression parameter data structure and the compressed data file.