Sign Language Recognition Using Gesture Fusion and Context Modeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional gesture language recognition systems achieve low accuracy due to the use of bracelets or gloves with sensors, making it difficult for ordinary people to effectively communicate with individuals with hearing impairments.
Innovation Solution
A method utilizing a terminal or terminal-server combination for gesture language recognition, employing two-dimensional and three-dimensional convolutional neural networks to extract gesture and change features, combined with long short-term memory networks for context-aware word recognition, without the need for additional devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional computer vision methods are used for sign language recognition, then the system structure is simple, but the recognition accuracy is low and real-time recognition is difficult to achieve
Solution Approach 1:
The patent segments the sign language recognition process into multiple specialized modules: hand region detection module, pose point detection module, skeleton construction module, and recognition module. Each module focuses on a specific aspect of the recognition task, improving overall accuracy while maintaining manageable system complexity through functional decomposition.
Solution Approach 2:
The patent introduces intermediate representations including pose points and skeleton structures that bridge the gap between raw image data and final recognition results. These intermediaries transform complex visual information into structured data that can be more effectively processed by recognition algorithms.
2Productivity
If traditional computer vision methods are used for sign language recognition, then the system structure is simple, but real-time recognition cannot be achieved
Solution Approach 1:
By dividing the recognition pipeline into parallel processing stages (hand detection, pose point detection, skeleton construction, recognition), the system can process multiple aspects of sign language simultaneously, improving real-time performance while keeping each individual module computationally efficient.
Solution Approach 2:
The system performs preliminary processing steps such as hand region detection and pose point identification before final recognition. This preliminary action prepares the data in advance, enabling faster real-time recognition by pre-processing critical features that will be needed for immediate classification.
3Adaptability or versatility
If existing recognition systems are used, then the system is simple to implement, but the recognition accuracy for various languages and scenarios is poor
Solution Approach 1:
The patent designs a universal recognition framework that can handle multiple sign languages (Chinese, English, French) and various scenarios (isolated signs, continuous speech, noisy environments) through a single integrated system. The skeleton-based representation and deep learning recognizer provide language-agnostic features that enable cross-lingual adaptability without requiring separate systems for each language.
Data Source
Figure 1
Figure 2
Figure 3(a)~3(c)
AI summary
The present application relates to a sign language recognition method and apparatus, a computer-readable storage medium, and a computer device. The method comprises: acquiring a sign language video on which recognition is to be performed; extracting a gesture feature from each frame of image in the sign language video; extracting a gesture change feature of each frame of image in the sign language video; extracting sign language word information from a fused feature obtained by fusing the gesture feature and the gesture change feature; and combining the sign language word information into a sign language sentence according to contextual information corresponding to the sign language word information.