Machine Learning System for Capturing Human Knowledge
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current training methods for complex tasks are inefficient and costly, as they rely on subject matter experts (SMEs) creating multimedia manuals or observing employees, which can omit crucial information and are not scalable for large groups or specialized tasks.
Innovation Solution
A machine learning system that captures human knowledge by correlating video, audio, and sensor data from SMEs to create a domain model, enabling the generation of targeted training information for novice users through augmented reality and interactive multimedia manuals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If SMEs create multimedia training manuals or conduct observations, then training quality can be maintained, but training time and costs increase significantly
Solution Approach 1:
The system creates digital copies of SME knowledge by capturing video, audio, and sensor data during task performance. These copies are processed through machine learning to generate training materials that replicate expert knowledge without requiring continuous SME involvement, thereby maintaining training quality while reducing time investment
Solution Approach 2:
The system enables automated knowledge capture where the training system itself collects and processes data during normal task performance. The machine learning algorithms automatically correlate multi-modal data and generate training materials without requiring dedicated training sessions, allowing the system to serve itself in knowledge extraction
2Loss of information
If SMEs create detailed multimedia training materials, then comprehensive knowledge coverage is achieved, but the complexity and cost of creation increase
Solution Approach 1:
The system merges multiple data sources (video, audio, sensor data) into a unified training model. By correlating these different modalities through machine learning, the system achieves comprehensive knowledge coverage while managing complexity through integrated processing rather than separate systems for each data type
Solution Approach 2:
The system transforms raw multi-modal data into structured training parameters and models. By changing the state of data from unprocessed multi-modal inputs to organized training materials, the system achieves comprehensive coverage while reducing the apparent complexity through parameter transformation
3Loss of information
If SMEs provide verbal descriptions during task demonstration, then implicit knowledge is captured, but the accuracy and completeness of knowledge transfer decrease
Solution Approach 1:
The system uses sensor data and video analysis to verify and correct the SME's verbal descriptions. By comparing actual task performance data with spoken explanations, the machine learning model identifies discrepancies and refines the knowledge representation, ensuring both implicit knowledge capture and high accuracy
Solution Approach 2:
The system creates a composite knowledge model that integrates multiple data types (video, audio, sensor readings) rather than relying solely on verbal descriptions. This composite approach captures implicit knowledge through non-verbal cues while maintaining accuracy through multi-source verification
4Ease of operation
If traditional training methods are used for large groups of employees, then individual attention can be provided, but scalability is limited
Solution Approach 1:
The system creates universal training models that can serve multiple employees simultaneously. The machine learning-generated training materials are reusable and adaptable to different learners while maintaining personalized learning paths, enabling the system to function as both a scalable automated system and an individualized training solution
Data Source
AI summary
This disclosure describes machine learning techniques for capturing human knowledge for performing a task. In one example, a video device obtains video data of a first user performing the task and one or more sensors generate sensor data during performance of the task. An audio device obtains audio data describing performance of the task. A computation engine applies a machine learning system to correlate the video data to the audio data and sensor data to identify portions of the video, sensor, and audio data that depict a same step of a plurality of steps for performing the task. The machine learning system further processes the correlated data to update a domain model defining performance of the task. A training unit applies the domain model to generate training information for performing the task. An output device outputs the training information for use in training a second user to perform the task.