Data Compression Using Bayesian Filtering of Complex Values
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing methods, such as those used in software like RAPIDMINER, often 'scrub' irrational and complex numbers from data sets, mistakenly identifying them as erroneous, which is problematic for fields like particle accelerator and cancer genomics where these numbers represent significant data.
Innovation Solution
A method and system that compress data sets by categorizing values into 'first' and 'second' categories, where values in the first category are retained and those in the second are excluded, using statistical distributions and Bayes' theorem to determine relevance, and storing in a quadtree data structure, ensuring complexities are preserved.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data scrubbing is performed to remove unexpected data, then data quality is improved, but significant data representing irrational and complex numbers is lost
Solution Approach 1:
The patent applies different processing rules to different types of data values. Instead of uniformly scrubbing all unexpected data, the system identifies and preserves specific categories of values (irrational numbers, complex numbers, mixed hashes, non-zero decimals beyond Xth place) while removing others. This localized quality approach maintains data reliability by selectively keeping significant data types.
Solution Approach 2:
The patent performs preliminary classification of data values into categories before the scrubbing process. By预先 identifying which values represent significant data (irrational numbers, complex numbers, etc.) and which are extraneous, the system avoids losing important information during compression and analysis operations.
2Loss of information
If all data values are retained in the data set, then data completeness is improved, but data processing efficiency deteriorates
Solution Approach 1:
The patent extracts and removes extraneous data values from the complete data set, keeping only significant categories (irrational numbers, complex numbers, mixed hashes, non-zero decimals beyond Xth place). This extraction process maintains data completeness for important values while improving processing efficiency by eliminating unnecessary data.
Solution Approach 2:
The patent changes the parameter of data representation by categorizing values based on their mathematical properties and significance. Instead of treating all values uniformly, the system applies parameter-based classification to determine which values to retain, achieving both completeness and efficiency.
3Productivity
If traditional data compression methods are used, then compression speed is improved, but complex and irrational numbers are mistakenly removed
Solution Approach 1:
The patent performs preliminary classification of data values into significant categories (irrational numbers, complex numbers, mixed hashes, non-zero decimals beyond Xth place) before compression. This preliminary action ensures that during rapid compression operations, the system knows which values to preserve, maintaining data accuracy while achieving compression speed.
Solution Approach 2:
The patent applies different compression rules to different data categories. Significant values (irrational numbers, complex numbers, etc.) are preserved with their original precision, while extraneous values are compressed or removed. This localized quality approach maintains compression speed while preventing erroneous removal of important data.
Data Source
AI summary
An example method of compressing a data set includes determining whether individual values from a data set correspond to a first category or a second category of values. Based on one of the values corresponding to the first category, the value is added to a compressed data set. Based on one of the values corresponding to the second category, the value is excluded from the compressed data set, and a statistical distribution of values of the second category is updated based on the value. During a first phase, the determining is performed for a plurality of values from a first portion of the data set based on comparison of the values to criteria. During a second phase, the determining is performed for a plurality of values from a second portion of the data set based on the statistical distribution.