Sign Language Captioning Model for Real-Time Timestamp Alignment

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Individuals with diminished hearing or reliance on sign language face challenges in interacting with others due to a lack of real-time translation capabilities and alignment of sign language, and existing devices often rely on microphones for voice commands, excluding other means for input.

Innovation Solution

A sign language captioning model is trained using a processor to generate training instances from sign language video content, incorporating feature tokens and ground truth captions, and deployed offline or online for real-time translation and alignment.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If real-time sign language translation capabilities are implemented, then communication accessibility for hearing-impaired users is improved, but device complexity and processing requirements increase

Engineering Contradiction:

Improvecommunication accessibilityVSAvoidprocessing requirements

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system performs preliminary actions by pre-processing sign language video content into structured training instances with extracted features (hand landmarks, bone structures, velocities) and corresponding captions before deployment. This pre-processing framework is established in advance to enable efficient real-time translation without requiring complex processing during actual communication events.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The sign language translation system is segmented into distinct functional modules: video input processing, hand landmark detection, bone structure extraction, velocity calculation, caption generation, and timestamp alignment. Each module handles a specific aspect of the translation process, reducing overall system complexity while maintaining comprehensive functionality.

Inventive Principle:

Principle #1Segmentation

2Measurement precision

If accurate timestamp alignment between sign language captions and video content is achieved, then translation precision is improved, but processing time and computational resources increase

Engineering Contradiction:

Improvecaption alignment accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system replaces traditional mechanical timestamp alignment methods with a learned mapping approach. The captioning model automatically generates timestamps aligned with sign language events through training on paired video-caption data, eliminating the need for manual or rule-based alignment mechanisms during operation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system changes parameters by transforming raw video data into extracted features (hand landmarks, bone structures, velocities) that capture essential sign language characteristics. These transformed parameters enable more efficient processing while maintaining alignment accuracy between captions and video content.

Inventive Principle:

Principle #35Parameter changes

3Adaptability or versatility

If multiple input modalities beyond microphone are provided for automated assistants, then user interaction flexibility is improved, but device complexity increases

Engineering Contradiction:

Improveinput modalitiesVSAvoidsystem architecture

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The automated assistant system is designed with multi-functionality to handle multiple input modalities including sign language video input, audio input, and text input through a unified architecture. The system can process different types of user inputs and generate appropriate responses, making the assistant universally accessible to users with different communication needs.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The system introduces an intermediary captioning model that translates sign language video input into text captions, which then serve as input to the existing automated assistant processing pipeline. This intermediary layer allows seamless integration of sign language support without fundamentally redesigning the core assistant architecture.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12562006B2System(s) and method(s) for training a sign language captioning model and subsequent use thereof

Publication Date: 2026.02.24 GOOGLE LLC

AI summary

Implementations are directed to training and subsequently utilizing a sign language captioning model. Initially, processor(s) of a system can obtain a plurality of training instances that are generated based on processing sign language video content, sign language conversations, etc. Each of the plurality of training instances can include at least corresponding sign language feature tokens for a sign language video content segment, ground truth caption tokens associated with ground truth sign language captions for the sign language video content segment and ground truth timestamp tokens that align the ground truth sign language captions with respect to the sign language video content segment. Further, the processor(s), can train the sign language captioning model based on the plurality of training instances, and can cause the sign language captioning model to be deployed in an offline manner and/or in an online manner for processing sign language content.