Dataset Integrity During User Interface Data Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inaccurate and outdated data in datasets can be time-consuming and costly to correct, especially in dynamic environments where data changes frequently, due to the inefficiencies in identifying and addressing inconsistencies.
Innovation Solution
A computer-implemented method that connects to a dataset, ingests data from a user interface, compares ingested data to expected data, and uses machine learning models to identify inconsistencies, with remedial actions such as flagging or replacing data, leveraging optical character recognition and web scraping for enhanced accuracy, and automating tasks with software robots.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual data cleaning and error correction is performed, then data accuracy can be maintained, but time consumption and cost increase significantly
Solution Approach 1:
The system performs data validation and consistency checking at the point of data ingestion, before the data enters the dataset. This preliminary action identifies and corrects errors automatically during the ingestion process, preventing time-consuming manual cleaning operations later while maintaining data accuracy.
Solution Approach 2:
The system uses automated software robots and machine learning models to perform data validation, anomaly detection, and error correction autonomously. The system serves itself by automatically identifying inconsistencies between ingested data and expected data, and executing remedial actions without requiring manual intervention, thus reducing time consumption while maintaining reliability.
2Productivity
If automated data ingestion is implemented, then productivity increases, but data integrity may deteriorate due to lack of manual verification
Solution Approach 1:
The system continuously monitors and compares ingested data against expected data patterns, backend DOM values, and dataset entries. This feedback mechanism automatically detects inconsistencies and triggers remedial actions, ensuring data integrity is maintained while automated ingestion processes continue to operate at high speed, thus resolving the contradiction between productivity and reliability.
Solution Approach 2:
The system performs validation checks during the data ingestion process itself, before data is stored. By conducting consistency checks between ingested data, expected data, and backend values at the point of ingestion, the system ensures data integrity is built-in during automated processing, eliminating the need for post-ingestion manual verification while maintaining high productivity.
3Measurement precision
If comprehensive data validation is performed, then data quality improves, but processing complexity and resource usage increase
Solution Approach 1:
The validation process is segmented into multiple independent checks: comparing ingested data against expected data patterns, validating against backend DOM values, checking consistency with existing dataset entries, and detecting anomalies through machine learning models. Each segmentation handles a specific aspect of validation, improving data quality while keeping individual processing steps manageable and modular, thus reducing overall processing complexity.
Solution Approach 2:
The system uses automated software robots and machine learning models to perform comprehensive validation autonomously. These self-service components handle complex validation logic without requiring manual intervention, reducing the operational complexity burden while maintaining high data quality standards. The automated nature of these processes eliminates the need for complex manual coordination and resource allocation.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach quickly identifies and corrects inconsistencies at the data ingestion stage, reducing manual effort and resource usage, particularly beneficial for datasets with frequent updates, by improving accuracy and efficiency in maintaining data integrity.
Implementation Method 1
performing optical character recognition of the UI and including recognized features from the optical character recognition as part of the expected data
Implementation Method 2
performing a web scraping of the UI and including data collected from the web scraping as part of the expected data
Data Source
AI summary
A computer-implemented method includes connecting to a dataset; ingesting data from a user interface (UI); and comparing ingested data in fields that have been entered to expected data. The expected data includes data in corresponding entries of the dataset, and backend document object model (DOM) values. The computer-implemented method further comprises identifying any inconsistencies between the ingested data and the expected data.