Multimodal Media Classification Using OCR and Audio Transcription

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing media content classification systems face challenges in efficiently and accurately analyzing both audio and video components of media items due to the need for extensive manual feature engineering and limitations in generalizing across diverse content types, leading to inconsistencies and errors.

Innovation Solution

A system that separately analyzes video and audio components of media items using optical character recognition (OCR) and audio transcription to generate feature vectors, which are then input into a classification-based machine learning model to determine the likelihood of the media item being associated with a predefined classification.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Extent of automation

If manual feature engineering is used for media content classification, then the system can process simple classification tasks, but it requires extensive manual effort and is prone to inconsistencies and errors

Engineering Contradiction:

Improveautomation of feature engineeringVSAvoidtime-consuming manual process

Core Design Contradiction:

Extent of automationVSLoss of time

Solution Approach 1:

The system performs automatic feature extraction from media content using machine learning models, eliminating the need for manual feature engineering. The models automatically learn relevant features from raw media data, enabling self-service classification without human intervention in the feature extraction process.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

Manual mechanical feature engineering is replaced with automated machine learning-based feature extraction. The system uses computational algorithms to automatically identify and extract features from media content, substituting human manual processes with automated computational systems.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Adaptability or versatility

If basic statistical methods are used for classification, then the system is simple to implement, but it is limited in generalizing across different types of content

Engineering Contradiction:

Improvegeneralization across content typesVSAvoidcomplexity of classification model

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system transitions from basic statistical parameters to sophisticated machine learning parameters and features. By changing the parameter representation and using advanced models, the system achieves better generalization across diverse content types while managing complexity through automated feature extraction.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The machine learning models are designed to handle multiple types of media content universally. The same classification framework and feature extraction processes work across different content types (images, videos, audio), providing multi-functional capability without requiring separate specialized systems for each content type.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Extent of automation

If deep learning models are used for automated classification, then manual feature engineering is reduced, but computational resources and processing time increase

Engineering Contradiction:

Improveautomated feature learningVSAvoidcomputational power consumption

Core Design Contradiction:

Extent of automationVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary feature extraction and processing before final classification. By pre-processing media content and extracting relevant features in advance, the system reduces the computational burden during actual classification operations, optimizing energy usage while maintaining automated feature learning capabilities.

Inventive Principle:

Principle #10Preliminary action

4Measurement precision

If comprehensive analysis of all media content is performed, then classification accuracy is improved, but processing efficiency decreases

Engineering Contradiction:

Improveclassification accuracyVSAvoidprocessing speed

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The system extracts only the most relevant features and information from media content for classification, rather than analyzing all aspects comprehensively. By selecting and extracting key features that are most indicative of content classification, the system maintains high accuracy while improving processing efficiency by avoiding unnecessary analysis of irrelevant details.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS20260030886A1Media classification system

Publication Date: 2026.01.29 TUBI INC

AI summary

A system and method for classifying media content, including: a computer processor and a video extraction and inference engine service executing on the computer processor and including functionality to obtain a video component and an audio component of a media item, perform optical character recognition (OCR) on a subset of frames of the video component, generate processed OCR text, and perform feature extraction on the processed OCR text to generate feature vectors representing the video component; an audio extraction and inference engine including functionality to transcribe the audio component to generate transcribed audio text, and perform feature extraction on the transcribed audio text to generate feature vectors representing the audio component; and a classification model serving engine configured to execute a classification-based machine learning model based on the feature vectors to generate a binary inference indicating the likelihood of the media item being associated with a predefined classification.