Subtitle Synchronization via Voice Timing Extraction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Television broadcast subtitles often lag behind the image display, causing synchronization issues and confusing audiences, especially in live broadcasts, and existing solutions require modifying broadcast standards and preparing both broadcast stations and reception devices, which is complex and difficult.

Innovation Solution

A data processing device and method that extracts character strings from subtitle data, searches voice data for corresponding utterances, generates subtitle timing information, and synchronizes subtitle and image data based on this information, allowing for synchronized display without modifying broadcast standards.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If subtitle data is transmitted through broadcast signal, then subtitles can be displayed with image data, but subtitles often lag behind image display causing synchronization issues

Engineering Contradiction:

Improvesubtitle-image synchronizationVSAvoidsubtitle delay

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary action by extracting character strings from subtitle data before synchronization, searching voice data to find corresponding utterances, and acquiring timing information in advance. This allows the reception device to pre-process and pre-align subtitle timing with image timing, eliminating the need for delay correction during actual display.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses feedback by comparing the timing information extracted from voice data with the timing information in subtitle data. The output control unit adjusts subtitle display timing based on this comparison, creating a closed-loop synchronization system that automatically corrects timing differences between subtitles and images.

Inventive Principle:

Principle #23Feedback

2Reliability

If delay time data transmission is implemented to correct subtitle lag, then subtitle synchronization can be improved, but broadcast standard modification and system preparation become complex and difficult

Engineering Contradiction:

Improvesubtitle synchronizationVSAvoidbroadcast standard preparation

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system applies self-service by enabling the reception device to autonomously extract timing information from voice data and automatically synchronize subtitle display without requiring external delay time data from the broadcast station. The reception device performs all synchronization operations independently using its own processing capabilities.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

Instead of having the broadcast station transmit delay time data to the reception device (traditional approach), this invention inverts the approach by having the reception device extract and determine timing information itself from the received voice and subtitle data, eliminating the need for broadcast standard modifications.

Inventive Principle:

Principle #13The other way round (Inversion)

3Adaptability or versatility

If manual subtitle production is used in live broadcasts, then subtitles can be created on the spot, but subtitles lag behind image display by several to tens of seconds

Engineering Contradiction:

Improvelive broadcast capabilityVSAvoidsubtitle delay

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system replaces the mechanical manual process of subtitle production with automated information processing. By using character string extraction from subtitle data and pattern matching against voice data to automatically determine timing, the system eliminates the manual typing and processing delays that cause tens of seconds of lag in live broadcasts.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS8781291B2Data processing device, data processing method, and program

Publication Date: 2014.07.15 SONY GROUP CORP

AI summary

A data processing device including an output control unit configured to extract a character string targeted for a search from subtitle data of content data including image data, voice data, and the subtitle data, search the voice data for data of a voice uttering the character string, acquire timing information indicating time when the character string is uttered, which is included in the image data, generate subtitle timing information including identification information provided to identify the subtitle data including the character string and the timing information, where the identification information and the timing information are associated with each other, so that the subtitle timing information is acquired, and synchronize the subtitle data and image data of time indicated by timing information associated with identification information of the subtitle data, and output the subtitle data and the image data based on the subtitle timing information is provided.