Field-State Pattern Coding for Faster Data Quality Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face challenges in efficiently analyzing and improving data quality by identifying patterns in large datasets, particularly in determining the state and validity of field populations across multiple records, which is crucial for data integrity and processing.
Innovation Solution
The method involves generating data pattern codes that represent the state or properties of fields or combinations of fields in records, using a set of element values to encode whether a field is populated, valid, or matches a target value, allowing for the analysis and summarization of data quality issues across datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is applied to reduce storage requirements, then storage efficiency is improved, but data access and processing complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing pattern codes that represent data patterns before actual data access. These pattern codes are generated in advance and stored alongside or instead of the full data, enabling rapid pattern identification without requiring complex real-time analysis of the compressed data structures.
Solution Approach 2:
The patent introduces pattern codes as an intermediary representation between the compressed data and the analysis processes. These codes serve as a simplified mediator that captures essential data patterns without requiring the full complexity of the original compressed data structures, thereby reducing processing complexity while maintaining analytical capability.
2Measurement precision
If detailed data analysis is performed to identify data quality patterns, then data quality assessment accuracy is improved, but processing time increases
Solution Approach 1:
The patent extracts essential data quality patterns by identifying and coding only the relevant characteristics of data patterns rather than analyzing entire data records. This extraction approach captures the critical information needed for quality assessment while eliminating unnecessary processing of redundant data, thereby maintaining accuracy while reducing processing time.
Solution Approach 2:
The patent transforms detailed data into simplified pattern codes that represent key data characteristics. This parameter transformation converts complex data structures into compact representations that retain essential quality information, enabling rapid analysis without sacrificing assessment accuracy.
3Device complexity
If pattern codes with fewer element values are used to represent data states, then code complexity is reduced, but information representation capability is limited
Solution Approach 1:
The patent merges multiple data characteristics into unified pattern codes that represent combinations of data states. By combining related data attributes into single coded elements, the system reduces code complexity while preserving the essential information about data patterns through aggregated representation.
Solution Approach 2:
The patent creates universal pattern codes that can represent multiple data states and conditions through a standardized coding scheme. These multi-functional codes can encode various data patterns using the same structural framework, reducing overall system complexity while maintaining comprehensive information representation capability.
Data Source
AI summary
A data storage system stores at least one dataset including a plurality of records. A data processing system, coupled to the data storage system, processes the plurality of records to produce codes representing data patterns in the records, the processing including: for each of multiple records in the plurality of records, associating with the record a code encoding one or more elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.