Definition Quality Scoring for Machine Learning Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data governance systems face challenges in creating and approving clear, unambiguous definitions for business assets, leading to contradictions and ambiguities in data interpretation and organization.
Innovation Solution
A system and method for automatically assessing the quality of definitions using machine learning models and guidelines, which evaluate structure, conciseness, circularity, and understandability, providing a score and allowing user feedback to improve definitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual definition creation and approval processes are used, then definitions can be carefully crafted and reviewed, but the process is time-consuming and prone to human error and inconsistency
Solution Approach 1:
The system performs preliminary automated evaluation of definitions against established quality guidelines before human review, pre-identifying issues with structure, conciseness, circularity, and understandability. This preliminary action filters out obviously defective definitions and prepares quality assessments in advance, reducing the time burden on manual reviewers while maintaining high quality standards.
Solution Approach 2:
The patent replaces manual mechanical review processes with automated machine learning models that evaluate definitions against quality guidelines. The ML system automatically assesses structural correctness, conciseness, circularity detection, and understandability metrics, substituting human mechanical review with computational analysis that operates faster and more consistently.
2Reliability
If multiple reviewers manually evaluate definitions, then comprehensive quality assessment is achieved, but resource consumption and coordination complexity increase
Solution Approach 1:
The quality assessment process is segmented into distinct evaluation dimensions: structure guidelines, conciseness metrics, circularity detection, and understandability measures. Each dimension is evaluated by specialized machine learning models, allowing comprehensive assessment without requiring multiple human reviewers to coordinate all aspects. The segmentation enables parallel processing of different quality aspects.
Solution Approach 2:
The machine learning evaluation system acts as an intermediary between definition creators and human reviewers. It provides automated quality scores and feedback that guide manual review efforts, reducing the burden on human reviewers while maintaining comprehensive quality assessment. The intermediary system consolidates multiple evaluation criteria into unified feedback.
3Productivity
If automated evaluation systems are implemented, then processing speed increases, but accuracy and nuance in quality assessment may decrease
Solution Approach 1:
The system applies different evaluation strategies to different aspects of definition quality: rule-based checks for structural correctness, statistical analysis for conciseness metrics, graph theory algorithms for circularity detection, and language modeling for understandability assessment. Each local aspect receives specialized treatment appropriate to its nature, maintaining precision while enabling automated high-speed evaluation.
Solution Approach 2:
The evaluation system combines multiple machine learning models and evaluation methods into a composite assessment framework. Different model types (rule-based, statistical, graph-based, language models) are integrated to evaluate various quality dimensions, creating a robust composite system that maintains high accuracy across diverse assessment criteria while operating at automated speeds.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system for preparing machine learning training data for use in evaluation of term definition quality. The system can include a server having at least one server processor and at least one server memory for storing a plurality of terms with corresponding definitions, and a plurality of client devices each having at least one client memory device and at least one client processor. The client processor programmed to receive at least one of the plurality of terms and its corresponding definition from the server, display the term and its corresponding definition, and receive an indication of whether the definition satisfies one or more definition quality guidelines. The server memory includes instructions for causing the at least one server processor to receive the indications from the plurality of client devices and label each definition as satisfying each of the definition quality guidelines or not based on the received indications.