Input File Validation Using Cross-Application Data Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data management systems struggle to compare, validate, and manage data from multiple applications operating in different formats, leading to inefficiencies and the need for manual intervention due to incompatible data formats and varying levels of granularity.
Innovation Solution
A workflow-based quality engineering automation solution that processes and stores data from multiple applications at a uniform level of granularity, applying rules to identify outliers, trends, and perform actions such as data deletion or validation, enabling direct comparison and visualization of data across applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data from multiple applications in different formats is processed directly, then data diversity and completeness are improved, but data comparability and processing efficiency deteriorate
Solution Approach 1:
The patent segments data processing into distinct phases: format detection, format identification, format conversion, and validation. Each phase handles specific aspects of heterogeneous data, allowing efficient processing of diverse formats without requiring manual intervention for each data type.
Solution Approach 2:
The patent introduces an intermediary data validation system that acts as a mediator between multiple applications with different data formats and the central processing system. This intermediary automatically detects, identifies, and converts formats, enabling seamless integration of diverse data sources without direct complex processing.
2Measurement precision
If manual validation and comparison of data from different formats is performed, then data accuracy is improved, but time consumption and operational complexity increase
Solution Approach 1:
The patent applies preliminary actions by automatically detecting and identifying data formats before validation and comparison operations. This preliminary format processing ensures that subsequent validation steps can proceed efficiently with standardized data, eliminating the need for manual format handling and reducing overall validation time.
Solution Approach 2:
The data validation system performs self-service by automatically detecting formats, identifying appropriate validation rules, and executing comparisons without human intervention. The system serves itself by maintaining format databases and automatically selecting validation methodologies based on detected data types.
3Measurement precision
If custom validation rules are created for each data format, then validation precision is improved, but system complexity and maintenance burden increase
Solution Approach 1:
The patent implements a universal validation framework that can handle multiple data formats through a single system architecture. The system identifies data formats and automatically applies appropriate validation rules from a centralized repository, eliminating the need for separate custom validation systems for each format while maintaining high validation precision.
Solution Approach 2:
The system manages complexity by parameterizing validation rules based on detected data formats. Instead of hardcoding separate validation logic for each format, the system dynamically selects and applies validation parameters based on format identification, allowing precise validation while keeping the underlying system architecture unified and maintainable.
Data Source
AI summary
A network system to analyze a combined output of various input files from data-based applications. The system provides custom profiling of the data from each application based on an application of one or more sets of rules. The system stores the data from any other number of applications in a base level of granularity to allow direct comparison of the data from each application output. Because the data is stored at a same level of granularity, the data may be compared or processed regardless of the application from which the data is received. The system applies rules to compare the data across the applications to identify outliers, trends, or commonalities. The system may also search for and identify data fitting a specific rule across the applications to extract, modify, or label the data. The system provides a visualization of the data based on the rules applied.