Mixed Input-Type Sequence Recognition Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing handwriting recognition systems struggle to effectively process and recognize mixed input-type sequences that combine both typeset symbols and handwritten strokes, which is common in interactive notetaking applications.
Innovation Solution
A method and computing device configuration that retrieve and process mixed input-type sequences by extracting typeset features, calculating geometric features for both typeset symbols and handwritten strokes, generating embeddings, and combining these embeddings to input a neural network for symbol recognition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a traditional handwriting recognition system processes only handwritten strokes, then the recognition accuracy for pure handwriting is maintained, but the system cannot effectively handle mixed input-type sequences combining typeset symbols and handwritten strokes
Solution Approach 1:
The system segments the input sequence into distinct typeset symbols and handwritten strokes, processing each type differently. Typeset symbols are extracted and processed through one pathway while handwritten strokes are processed through another, allowing specialized handling of each input type to maintain accuracy while expanding versatility
Solution Approach 2:
The recognition system is designed to handle multiple input types (pure handwriting, pure typeset, and mixed sequences) through a unified architecture that adapts its processing based on input type detection, making the system universally applicable to various notetaking scenarios
2Measurement precision
If the system extracts and processes multiple feature sets for typeset symbols and handwritten strokes, then the recognition accuracy improves, but the computational complexity and processing time increase
Solution Approach 1:
Different feature extraction methods are applied locally to different input types: typeset symbols receive one type of feature processing while handwritten strokes receive another, optimizing each pathway for its specific input type without unnecessarily complicating the overall system
Solution Approach 2:
The system dynamically adjusts processing parameters based on input type detection, applying appropriate feature extraction and embedding dimensions for typeset versus handwritten inputs, thereby maintaining accuracy while managing computational complexity through adaptive parameter selection
Data Source
Figure 1
Figure 2
Figure 3A~3C
AI summary
A system and a method for recognizing a mixed input-type sequence, wherein the mixed input-type sequence comprises at least a typeset symbol and at least a handwritten stroke, comprising: retrieving the at least one typeset symbol and the at least one handwritten stroke (ST4-ST6); extracting a set of typeset features (STF1-STF3) for the at least one retrieved typeset symbol; calculating a first set of geometric features (SGF1-SGF3) for the at least one retrieved typeset symbol and a second set of geometric features (SGF4-SGF6) for the least one retrieved handwritten stroke; generating a typeset embedding (TE1-TE3) from the extracted set of typeset features (STF1-STF3) and assigning a typeset embedding (TEc) for the at least one retrieved handwritten stroke; generating geometric embeddings (GE1-GE6) from the calculated sets of geometric features; combining the generated typeset embedding (TE1-TE3) of the at least one typeset symbol and the assigned typeset embedding (TEc) of the at least one handwritten stroke with the respective geometric embeddings; applying the combined embeddings (CE1-CE6) as an input sequence representing the mixed input-type sequence to a trained neural network (NNC); and generating a sequence (OS) of recognized symbols.