Online Meeting Context Inference via Non-Verbal Video Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current online meeting technologies lack a comprehensive and efficient way to capture and summarize meeting content, as text-based transcripts derived from speech-to-text algorithms are incomplete and do not account for non-verbal communications, making it difficult for participants to gain accurate insights into what occurred during a meeting.
Innovation Solution
A system that uses machine learning and computer vision to generate a complete digital representation of online meetings by processing video streams to extract text from spoken messages and non-verbal communications, which is then used to provide a more accurate input for a generative language model to generate summary descriptions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If speech-to-text algorithms are used to generate meeting transcripts, then text-based meeting records can be obtained, but non-verbal communications are not captured making the transcripts incomplete
Solution Approach 1:
The patent combines speech-to-text transcription with computer vision-based non-verbal communication detection into a unified meeting analysis system. The video processing module and audio transcription module work together to generate comprehensive meeting transcripts that include both verbal and non-verbal communications, resolving the information loss problem while managing system complexity through integrated processing.
Solution Approach 2:
The meeting analysis service is designed to perform multiple functions: transcribing speech, detecting non-verbal communications (gestures, facial expressions), extracting text from shared content, and generating contextualized meeting summaries. This multi-functional approach captures complete meeting content while providing a comprehensive solution that addresses various aspects of meeting analysis in a single system.
2Loss of information
If video streams are processed to extract non-verbal communications, then more complete meeting information is obtained, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary processing of video streams during the meeting by detecting and analyzing non-verbal communications in real-time or near real-time. By preprocessing the video data and extracting key non-verbal cues during the meeting rather than analyzing entire video files afterward, the system reduces the time required for post-meeting summary generation while maintaining accurate representation of the meeting content.
Solution Approach 2:
The patent extracts only the essential non-verbal communication elements (key gestures, significant facial expressions) from the video stream rather than processing and analyzing every frame in detail. This selective extraction approach captures the most relevant non-verbal information while minimizing computational overhead and processing time, balancing accuracy with efficiency.
3Loss of information
If text from shared content is extracted and combined with transcripts, then comprehensive meeting context is achieved, but data processing complexity increases
Solution Approach 1:
The patent merges text extracted from shared content (presentations, documents, screenshares) with the speech-to-text transcript and non-verbal communication data into a unified context. The meeting analysis service integrates these multiple text sources, correlating them with timestamps and contextual information to create a comprehensive meeting record that captures both the discussion and the visual materials presented, achieving complete meeting context through combined processing.
Data Source
AI summary
A system for deriving a digital representation of an online meeting using contextual data inferred from non-verbal communications. The system processes video streams from meeting participants, including content shared during the meeting. Object detection algorithms identify regions of interest in shared content, such as text collections or graphics. For text collections, layout analysis algorithms determine the structure, enabling extraction of word groups. These word groups are stored with timestamps indicating when they were shared. The system can also detect gestures made by participants, generate textual descriptions of the gestures, and integrate them with a text-based transcript of the meeting. This creates a chronological sequence of both verbal and non-verbal communications. In response to user queries, the system generates responses based on the extracted word groups, gesture descriptions, and their corresponding timestamps, providing comprehensive insights into the online meeting.