Unified Data Validation With Parallel Cross-Format Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional big data systems face challenges in managing and validating terabytes of data due to proprietary environments, data format variations, and increasing complexity, leading to inefficient resource usage and error-prone manual configurations.
Innovation Solution
A data management system with a data compare system that utilizes configuration properties and data source configurations to automate data comparison and validation across different formats, leveraging parallel processing and machine learning to reduce errors and improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional database systems are used to manage terabytes of data from different sources, then data storage capacity is sufficient, but data validation becomes error-prone due to proprietary environments and format variations
Solution Approach 1:
The patent introduces a data validation service as an intermediary component that sits between data sources and traditional database systems. This service standardizes data validation across multiple proprietary environments by implementing a unified validation framework that handles format variations and cross-system compatibility issues, thereby improving validation reliability without increasing overall system complexity
Solution Approach 2:
The validation service implements universal validation rules and procedures that can handle multiple data formats and sources through a single standardized interface. This multi-functional approach allows the same validation logic to work across different proprietary database systems and data formats, reducing the need for system-specific validation code and improving overall reliability
2Productivity
If manual configuration methods are used for data validation, then system complexity is low, but processing time increases and errors occur
Solution Approach 1:
The validation service implements self-service capabilities through automated rule generation and configuration. The system can automatically discover data patterns, generate validation rules, and configure validation parameters without requiring extensive manual intervention. This automation significantly reduces both processing time and the potential for human errors while maintaining low operational complexity
Solution Approach 2:
The system performs preliminary validation actions by pre-configuring validation rules and preparing validation frameworks before actual data processing begins. This advance preparation includes setting up validation schemas, defining data type mappings, and establishing error handling procedures, which accelerates the actual validation process and reduces overall processing time
3Measurement precision
If data formats from different sources are standardized, then validation accuracy improves, but processing time increases
Solution Approach 1:
The validation service dynamically adjusts data format parameters based on the specific validation requirements and data source characteristics. Instead of uniformly converting all data to a single format, the system selectively transforms only the parameters that are relevant to validation accuracy, preserving original formats where they don't impact validation precision. This selective parameter transformation maintains high comparison accuracy while minimizing processing overhead
Data Source
AI summary
Examples described herein include implementations for big-data validation. One aspect includes generating a configuration file including dynamic matching data describing a first plurality of data entries and a second plurality of data entries, and generating a data action file. A plurality of data queries are generated based on the dynamic matching data indicated in the configuration file. The plurality of data queries are dynamically executed in parallel, including execution of a plurality of simultaneous data queries to the data source system. Fields of the first plurality of data entries and the second plurality of data entries are matched using the key type and the value structure, corresponding fields of the first data fields and the second data fields having a data mismatch are identified, and a mismatch database entry for the corresponding fields having the data mismatch is automatically generated.