Local Audio Captioning With Metadata-Based Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Most audible content is not captioned, making it inaccessible to users with hearing difficulties, and existing automatic captioning systems are expensive, complex, require audio output, or compromise user privacy and convenience.
Innovation Solution
A computing device generates captions locally using a machine-learned model that analyzes original audio data, determines suitability for captioning based on metadata, and provides descriptions visually without audio output, ensuring privacy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If cloud-based captioning services are used to avoid costs and complexities, then device complexity and cost are reduced, but user privacy and convenience are compromised
Solution Approach 1:
The patent introduces an audio mixer as an intermediary component that captures audio data at the system level before it reaches applications. This mediator enables local captioning processing without requiring cloud services, thereby reducing device complexity while maintaining user privacy and convenience through offline operation.
Solution Approach 2:
The computing device performs captioning services itself using locally-hosted machine-learned models rather than relying on external cloud services. This self-service approach eliminates the need for internet connectivity, protects user privacy by keeping data local, and provides offline convenience while maintaining system simplicity.
2Measurement precision
If machine-learned model analyzes all audio data to ensure accuracy, then captioning accuracy is improved, but processing time and computational resources increase
Solution Approach 1:
The patent applies preliminary filtering by analyzing metadata (such as content type identifiers) before processing audio data through the full machine-learned model. This preliminary action identifies and filters out audio data that is unlikely to contain captionable content (e.g., music, notifications), reducing processing time while maintaining accuracy for relevant audio streams.
Solution Approach 2:
Instead of applying the complete machine-learned model to all audio data, the system applies partial processing by first filtering based on metadata and only then applying the full model to selected audio streams. This partial action approach reduces computational overhead and processing time while maintaining captioning accuracy for appropriate content.
3Ease of operation
If audio output to speaker is used as input for captioning, then captioning can be provided, but the system is less suited for quiet environments and requires audible sound signals
Solution Approach 1:
The patent transitions from using acoustic dimension (audio output to speaker) to a different dimension by directly accessing audio data at the system level through an audio mixer. This dimensional change allows captioning to operate without requiring audible sound output, enabling functionality in quiet environments and eliminating the need for speaker output as an input source.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computing device is described for automatically captioning content. The computing device generates captions directly from audio data being output from content sources, unlike other captioning systems which often rely on information contained in audio signals being sent to speakers. The computing device may analyze metadata to determine whether the audio data is suitable for captioning or whether the audio data is some other type of audio data. Responsive to identifying audio data for captioning, the computing device can quickly generate a description of audible sounds interpreted from the audio data. The computing device can provide the descriptions of audible content visually no matter the application source of the audible content and regardless whether the computing device ever produces audio of the content.