Multi-Modal Drawing Prediction Using Feature Concatenation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current electronic devices are unable to effectively classify drawings into emojis or stickers using user strokes due to limitations in handling multi-modal inputs and require complete strokes for accurate prediction, leading to cumbersome user experiences and inefficiencies.
Innovation Solution
A method and electronic device that segregate features from text input and user strokes into dominant and non-dominant features, perform concatenation using a deep neural network (DNN) model, and predict multi-modal drawings with minimal user strokes, enabling efficient classification and prediction of emojis and stickers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complete user strokes are required for accurate drawing classification, then classification accuracy is improved, but user effort and operation time increase
Solution Approach 1:
The system performs preliminary classification using available strokes before the user completes the entire drawing. By analyzing partial stroke data and predicting potential drawings early in the input process, the system provides suggestions without requiring complete user input, thus maintaining accuracy while reducing effort
Solution Approach 2:
The system uses only a portion of the complete stroke sequence for initial classification rather than waiting for all strokes. By processing partial action data and generating predictions incrementally, the system achieves accurate classification with minimal user input
2Device complexity
If multi-modal features (text and drawing) are processed separately, then feature processing simplicity is maintained, but prediction accuracy deteriorates
Solution Approach 1:
The system merges text features and drawing stroke features into a unified multi-modal feature representation. By concatenating and jointly processing these different feature types through the neural network, the system captures interactions between text and drawing modalities, significantly improving prediction accuracy
Solution Approach 2:
The system creates a composite feature vector that combines heterogeneous data types (text embeddings and stroke features) into a unified representation. This composite approach allows the neural network to leverage complementary information from multiple modalities simultaneously
3Adaptability or versatility
If traditional handwriting engine is used without emoji scope utilization, then compatibility with existing systems is maintained, but emoji prediction capability is lost
Solution Approach 1:
The system enhances the traditional handwriting engine to serve multiple functions: it processes both conventional text input and emoji/drawing prediction. By integrating emoji scope utilization into the existing engine architecture, the system maintains backward compatibility while adding new automated emoji prediction capabilities
Solution Approach 2:
The system enables the handwriting engine to automatically predict emojis by leveraging its own feature processing capabilities and integrated neural network, without requiring separate dedicated systems. The engine self-adapts to handle both text and emoji prediction tasks
Data Source
AI summary
Embodiments herein disclose a method and electronic device for predicting multi-modal drawings. The method includes: receiving, by the electronic device, at least one of a text input and strokes of a drawing and determining, by the electronic device, features associated with the text input and features associated with the strokes of the drawing. The method includes classifying, by the electronic device, the features associated with the text input and the features associated with the strokes of the drawing into one of a dominant feature and a non-dominant feature and performing, by the electronic device, early concatenation or late concatenation of the features based on the classification; classifying, by the electronic device, the strokes of the drawing based on the concatenation into a category using a deep neural network (DNN) model; and predicting, by the electronic device, primary drawings corresponding to the category.