Crowdsourced NLP Training via Unmanaged Voice Data Collection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Natural Language Processing (NLP) systems face challenges in accommodating the wide variety of utterances present in a given language, particularly in recognizing different pronunciations, syntaxes, and word orders used by diverse end-users for similar tasks, making it difficult to collect and store high-quality text commands effectively.
Innovation Solution
The system employs unmanaged crowds to generate entity-level annotated text utterances through variant elicitation tasks, domain validation, and spellchecking, using crowdsourcing platforms to collect and validate text commands, and employs non-machine readable captions to prevent automated responses, allowing for the collection of high-quality text variants.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional NLP systems use traditional training methods with managed crowds in recording studios, then the quality of voice data capture is maintained, but the cost and time required for data collection increases significantly
Solution Approach 1:
The patent uses unmanaged crowds of ordinary users who provide voice data through mobile devices, replacing the expensive managed crowd model. These users are not professionally trained and provide data casually, significantly reducing costs while maintaining sufficient data quality for training NLP systems
Solution Approach 2:
The patent replaces the mechanical recording studio setup with automated mobile application-based data collection. The system uses smartphone keyboards and microphones to capture voice and text data, eliminating the need for physical recording facilities and professional equipment
2Adaptability or versatility
If NLP systems attempt to recognize all variations of user utterances, then the system becomes more adaptable to different users, but the complexity of training data collection increases
Solution Approach 1:
The patent creates a universal data collection platform that works across different devices and user types. The mobile application serves multiple functions: capturing voice, transcribing to text, collecting contextual information, and validating data quality, all within a single system that adapts to various users without requiring specialized training
Solution Approach 2:
The system enables ordinary users to contribute training data through their own mobile devices without requiring professional trainers or complex setup. Users naturally provide voice and text data through normal smartphone usage, and the system automatically processes and validates this self-provided data
3Productivity
If the system uses unmanaged crowds for data collection, then the cost and accessibility of data collection improves, but the quality control and validation of collected data becomes more difficult
Solution Approach 1:
The patent implements multiple feedback loops for quality control: automated transcription validation, contextual consistency checks, and human reviewer verification for borderline cases. The system continuously monitors data quality metrics and adjusts collection parameters accordingly, ensuring reliable data from unmanaged crowds
Solution Approach 2:
The patent introduces automated text processing and validation systems as intermediaries between unmanaged crowd inputs and the training dataset. These intermediaries filter, validate, and standardize user-provided data before it reaches the training pipeline, maintaining quality control without requiring direct human management of each data point
Data Source
AI summary
Systems and methods gathering text commands in response to a command context using a first crowdsourced are discussed herein. A command context for a natural language processing system may be identified, where the command context is associated with a command context condition to provide commands to the natural language processing system. One or more command creators associated with one or more command creation devices may be selected. A first application one the one or more command creation devices may be configured to display command creation instructions for each of the one or more command creators to provide text commands that satisfy the command context, and to display a field for capturing a user-generated text entry to satisfy the command creation condition in accordance with the command creation instructions. Systems and methods for reviewing the text commands using second and crowdsourced jobs are also presented herein.