Interactive Data Quality Dashboard for Iterative ML Data Cleaning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data quality management tools lack automation in data profiling, validation, and correction, are often outdated, dependent on user understanding, have limited scope, and are complex to configure and integrate, leading to inefficiencies in managing data quality for machine learning applications.
Innovation Solution
An automated, interactive data quality dashboard that leverages advanced algorithms for data profiling, validation, and correction, integrating with ML tools like MLflow and Delta Lake, supports iterative cleaning, and includes a user-in-the-loop strategy for rule definition and validation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated algorithms are used for data profiling, validation, and correction, then productivity and data quality are improved, but device complexity increases
Solution Approach 1:
The system is divided into distinct modular components: data profiling module, validation module, correction module, and ML model integration module. Each module performs a specific function independently, allowing the complex data quality management process to be broken down into manageable segments that can be developed, maintained, and scaled separately while working together as an integrated system.
Solution Approach 2:
The patent introduces intermediate layers between raw data and ML models, including data quality dashboards, validation rules engines, and correction algorithms. These intermediaries act as mediators that automatically process, validate, and clean data before it reaches the ML models, reducing the technical burden on users while maintaining system effectiveness.
2Reliability
If comprehensive data quality management is implemented, then data quality improves, but ease of operation decreases
Solution Approach 1:
The system implements self-service capabilities through automated data profiling that independently analyzes data characteristics, validation rules that automatically check data quality, and correction algorithms that autonomously fix identified issues. The system serves itself by automatically detecting and correcting data quality problems without requiring extensive user intervention or deep technical knowledge.
Solution Approach 2:
The patent incorporates feedback mechanisms where the system continuously monitors data quality metrics, compares them against predefined standards, and automatically adjusts correction strategies based on the results. This closed-loop feedback system ensures high data quality while minimizing user burden, as the system learns from and adapts to data patterns automatically.
3Loss of time
If automated error detection and correction is implemented, then loss of time is reduced, but manufacturing precision requirements increase
Solution Approach 1:
The system performs preliminary data profiling and validation before main data processing operations. By pre-identifying potential data quality issues, validation rules, and correction opportunities upfront, the system reduces the time required for subsequent data cleaning while maintaining high precision through pre-configured validation criteria and correction algorithms.
Data Source
AI summary
Certain example embodiments relate to dashboards that help streamline and automate data quality management processes used with machine learning (ML) models and ML-enabled technology. A clean dataset is initialized from a dirty dataset. A search space is the set of all possible combinations of available error detection algorithms and data repair algorithms. A scoring function measures performance of a given error detection algorithm and data repair algorithm combination on the clean dataset. An ML model is trained using the clean dataset. Best error detection and data repair algorithms are selected, based on an optimization on the set of all possible combinations, and the defined scoring function. The selected best error detection algorithm is applied to the clean dataset, and a repaired dataset is generated using the selected best repair algorithm. The clean dataset is set to the repaired dataset. This procedure is repeated until a condition is met.