Runtime Data Validation Rules Engine for Mobile Source Cleansing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data validation processes are inefficient and costly, often requiring multiple steps and batch processing, and are unable to manage complex inter-related data scenarios or update data structures and validation rules in real-time, leading to incomplete and inaccurate data.
Innovation Solution
A system and method that utilize generic configurable processes, such as a data API and validation rules engine, to manage unknown data structures and validation rules at runtime, enabling data cleansing and quality control directly on mobile devices, allowing for real-time data processing and updates without changing the source code of applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is collected from multiple sources and sent to a cleaning site for batch processing, then data cleaning can be performed, but the process is time-consuming and expensive
Solution Approach 1:
The patent implements preliminary data validation and cleaning rules at the data collection source (mobile devices) before data is transmitted to the server. Configuration files containing validation rules are downloaded to mobile devices, which then perform immediate data verification, error detection, and correction at the point of data entry, eliminating the need for time-consuming batch processing later
Solution Approach 2:
The patent introduces configuration files as an intermediary layer between data collection and data processing. These configuration files contain validation rules, data structure definitions, and cleaning parameters that mediate the data processing workflow, enabling automated real-time validation at the source while maintaining the ability to perform comprehensive batch processing when needed
2Reliability
If conventional batch processing is used for data cleaning, then cleaning can be performed in off-periods, but the source of data is not available to provide corrections
Solution Approach 1:
The patent enables mobile devices to perform self-service data validation and correction by downloading configuration files containing validation rules and data structure definitions. The mobile devices automatically validate their own data entries against these rules, detect errors, and correct them locally without requiring server-side intervention or batch processing, making the data source itself capable of ensuring data quality
Solution Approach 2:
The patent implements a feedback mechanism where validation results and error detections from mobile devices are transmitted back to the server. The server can then provide corrective feedback by updating configuration files with refined validation rules based on accumulated error patterns, creating a continuous improvement loop that enhances data quality over time
3Ease of operation
If regular expressions are used for simple validation, then basic data entry errors can be prevented, but complex inter-related data scenarios cannot be managed
Solution Approach 1:
The patent transitions from static regular expression validation to dynamic, multi-layered validation rules stored in configuration files. These rules can be updated and modified without changing application code, allowing the validation system to adapt to complex, inter-related data scenarios while maintaining ease of operation through a structured rule-based approach that can handle both simple and complex validation requirements
4Reliability
If data structures and validation rules are hard-coded in applications, then specific use cases can be addressed, but updates require source code changes and new application versions
Solution Approach 1:
The patent segments the validation system into two independent components: the application code (which remains stable) and the configuration files (which contain data structures and validation rules). This segmentation allows validation rules to be updated by simply distributing new configuration files without requiring application code changes or new version deployments, reducing software complexity while maintaining validation consistency
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
Apparatus and method to cleanse data, the apparatus including: a receiver to collect electronic data to cleanse; a processor coupled to the receiver, the processor configured to receive the data from the receiver; a memory coupled to the processor, the memory configured to store an application program; a first interface to an instantiation module, to process data collected by the receiver; and a second interface to a configuration manager module, the configuration manager module configured to control data structure and rules used by the instantiation module to process data, wherein the first interface and the second interface are callable from the application program to cleanse the data collected by the receiver.