Variable Length Embedding Table for Predictive Text
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network-based data entry systems on resource-constrained devices, such as smartphones and smartwatches, face challenges due to high memory and processing resource requirements, leading to inefficiencies in predictive text input, where accuracy and performance are compromised.
Innovation Solution
Implementing a variable length item embedding table within these devices, where item embeddings for frequent and infrequent inputs differ in length, and using a projector to adjust embedding lengths for neural network input and output stages, reducing memory usage while maintaining prediction quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural network technology is used for predictive text input, then prediction accuracy is improved, but memory and processing resource requirements increase
Solution Approach 1:
The patent segments the embedding table into multiple sub-tables, each storing embeddings of a specific data type (e.g., nouns, verbs, adjectives). This segmentation reduces the memory footprint of each individual sub-table while collectively maintaining comprehensive coverage of the vocabulary space, thus resolving the contradiction between prediction accuracy and memory usage.
Solution Approach 2:
The patent applies local quality by using different embedding dimensions for different data types. Specifically, certain data types are represented with higher-dimensional embeddings that capture more nuanced semantic information, while other data types use lower-dimensional embeddings. This selective approach maintains prediction accuracy for critical data types while reducing overall memory consumption.
2Measurement precision
If neural network technology is used for predictive text input, then prediction accuracy is improved, but processing resources increase
Solution Approach 1:
The patent segments the embedding lookup process into multiple independent sub-table lookups based on part-of-speech tags. This segmentation allows the neural network to process different data types through specialized, optimized pathways, reducing the overall computational burden compared to a single large embedding table, thus resolving the contradiction between prediction accuracy and processing resources.
Solution Approach 2:
The patent changes the parameter of embedding dimensionality based on the data type being processed. By using variable embedding dimensions tailored to specific data types (e.g., lower dimensions for common function words, higher dimensions for content words), the system optimizes the balance between prediction accuracy and processing resource consumption.
3Quantity of substance
If variable length item embeddings are used, then memory usage is reduced, but embedding projection complexity increases
Solution Approach 1:
The patent applies local quality by using different embedding lengths for different data types within the segmented embedding tables. Each sub-table contains embeddings of uniform length appropriate for its data type, simplifying the projection process within each segment while achieving overall memory reduction through the variability across segments.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A data entry system is described which has a user interface which receives a sequence of one or more context text items input by a user. The data entry system has a predictor trained to predict a next item in the sequence. The predictor comprises a plurality of learnt text item embeddings each text item embedding representing a text item in a numerical form, the text item embeddings having a plurality of different lengths. A projection component obtains text item embeddings of the context text items and projects these to be of the same length. The predictor comprises a trained neural network which is fed the projected text item embeddings and which computes a numerical output associated with the predicted next item.