Data Pattern Coding for Field Correlation and Quality Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data quality measurement methods are inadequate in analyzing field-level data patterns and correlations within datasets, particularly in identifying mandatory and conditionally mandatory fields, and in detecting data quality issues and hidden business rules across diverse data sources.
Innovation Solution
The method involves generating data pattern codes that summarize the state of field population and validity, enabling the identification of mandatory and conditionally mandatory fields, and detecting data quality issues by analyzing correlations between fields, which can be used to infer business rules and validate data against predefined schemas.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in compressed format to reduce storage requirements, then storage efficiency is improved, but data access and analysis complexity increases
Solution Approach 1:
The patent pre-computes and stores data pattern codes during data ingestion, before analysis is needed. These pattern codes capture field population states, correlations, and business rules in advance, so when compression is applied, the pre-extracted pattern information remains accessible without requiring full decompression for analysis.
Solution Approach 2:
The patent introduces data pattern codes as an intermediary representation between the compressed raw data and the analysis queries. These codes serve as a mediator that encapsulates essential data characteristics, allowing analysis to proceed on the compact pattern representation rather than requiring full decompression of compressed data.
2Measurement precision
If field-by-field data quality analysis is performed to identify mandatory fields and correlations, then data quality measurement precision is improved, but analysis time and computational resources increase
Solution Approach 1:
The patent segments the data quality analysis into distinct components: field population state analysis, correlation analysis, and business rule inference. Each component generates specific pattern codes that can be computed independently and combined, allowing parallel processing and reducing overall analysis time while maintaining comprehensive coverage.
Solution Approach 2:
The patent performs preliminary analysis to identify and encode data patterns, correlations, and business rules during data ingestion or in a pre-processing phase. This upfront work captures essential relationships so that subsequent data quality measurements can be performed quickly by referencing pre-computed patterns rather than re-analyzing raw data from scratch.
3Reliability
If comprehensive data pattern codes are generated to capture all field states and correlations, then data quality detection capability is improved, but data processing complexity increases
Solution Approach 1:
The patent segments comprehensive data pattern analysis into manageable components: individual field state coding, pairwise correlation detection, and multi-field relationship identification. Each segment produces intermediate pattern codes that are progressively combined, making the overall complex task tractable through systematic decomposition.
Solution Approach 2:
The patent transforms complex multi-field correlation analysis into a different dimensional representation using bitmaps and pattern codes. Instead of analyzing relationships between fields in the original data space, the system projects these relationships into a compressed pattern space where correlations are encoded as bit patterns, simplifying the detection and analysis of complex relationships.
4Measurement precision
If data pattern codes are used to identify hidden business rules through field correlations, then business rule detection accuracy is improved, but computational requirements increase
Solution Approach 1:
The patent performs preliminary correlation analysis to identify potential business rules during data ingestion or in a separate preprocessing phase. By detecting field correlations and encoding them as patterns in advance, the system reduces the computational burden during actual business rule detection, as the heavy lifting of correlation analysis has already been performed.
Solution Approach 2:
The patent applies partial correlation analysis by focusing on the most significant field relationships rather than computing all possible correlations. It identifies and encodes the subset of correlations that are most likely to represent business rules, based on heuristics such as correlation strength, field importance, and domain knowledge, thereby reducing computational requirements while maintaining detection accuracy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A data storage system (102) stores at least one dataset including a plurality of records. A data processing system (104), coupled to the data storage system, processes the plurality of records to produce codes (114) representing data patterns in the records, the processing including: for each of multiple records in the plurality of records, associating with the record a code encoding one or more elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.