Data Annotation Pair Refinement for Multi-Intent Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional feature extraction techniques are hindered by data heterogeneity, and existing approaches fail to determine whether provided annotations are sufficient to convey an intent, leading to insufficient learning of generalizable programs.
Innovation Solution
A method that analyzes annotation properties, identifies non-diffs and multiple intents, generates new annotations, and incorporates user feedback to improve data annotation quality by modifying and merging annotations based on identified properties and feedback.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional feature extraction techniques are used, then data processing can be performed, but data heterogeneity reduces the quality and generalizability of the extracted features
Solution Approach 1:
The system transforms heterogeneous data by applying parameter changes through learned transformation programs that convert source columns into target columns, adapting the data to a homogeneous format suitable for feature extraction
Solution Approach 2:
The system introduces an intermediary data transformation layer that mediates between heterogeneous source data and the feature extraction process, using learned transformation programs to bridge the heterogeneity gap
2Productivity
If a limited number of annotation pairs are provided, then annotation processing is faster and requires less resources, but the annotations are insufficient to learn generalizable programs
Solution Approach 1:
The system performs preliminary analysis of annotation pairs to identify properties associated with multiple intents before proceeding with program learning, ensuring that sufficient and high-quality annotations are available upfront
Solution Approach 2:
The system incorporates feedback loops that analyze annotation quality and provide information about whether the number of annotations is sufficient to convey intent, enabling iterative improvement of annotation sets
3Device complexity
If conventional approaches are used, then simple annotation processing can be performed, but they fail to determine whether a given number of annotations are sufficient to convey an intent
Solution Approach 1:
The system replaces conventional mechanical annotation processing with an intelligent system that uses property-related rules and automated analysis to determine annotation sufficiency and identify multiple intents
4Reliability
If data transformation is performed to convert heterogeneous data into homogeneous data, then data quality improves, but additional processing steps and computational resources are required
Solution Approach 1:
The system performs self-service data transformation by automatically learning transformation programs from example input-output pairs, eliminating the need for manual transformation rule creation and reducing overall system complexity
Data Source
AI summary
Methods, systems, and computer program products for automatically improving data annotations by processing annotation properties and user feedback are provided herein. A computer-implemented method includes obtaining data annotation pairs, each comprising an input data annotation in a first format and a corresponding output data annotation in a second format; determining, within at least a portion of the data annotation pairs, one or more non-diffs; identifying, across the at least a portion of data annotation pairs, data annotation properties associated with multiple intents by processing the non-diffs using property-related rules; modifying at least a portion of the data annotation pairs based on the identified data annotation properties; outputting the modified data annotation pairs to at least one user; and generating a final collection of data annotation pairs by processing at least a portion of the modified data annotation pairs and user feedback received in response to the outputting.