Text Processing Method Retaining Numerical Information
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language processing methods for medical text often lose numerical information during pre-processing, as embeddings are only learned for frequently occurring tokens, leading to non-informative or missing numerical data, which can result in loss of crucial information for deep learning models.
Innovation Solution
A method that transforms numerical tokens into common encoding vectors and assigns respective numerical vectors based on their values, extending embedding vectors to retain numerical information, allowing these extended vectors to be used as input for deep learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If embeddings are learned only for frequently occurring tokens, then the deep learning model can process common words effectively, but numerical information is lost when numbers occur infrequently
Solution Approach 1:
The patent segments the token representation into two distinct components: a common embedding vector (capturing semantic meaning) and a numerical vector (capturing numerical properties). This segmentation allows each component to specialize - the embedding handles semantic similarity while the numerical vector preserves exact numerical values, thereby preventing information loss for infrequent numerical tokens.
Solution Approach 2:
The patent extends the traditional embedding vector by appending a numerical vector component, creating an extended embedding in a higher-dimensional space. This dimensional extension allows the system to simultaneously represent both semantic information (from the original embedding) and numerical information (from the appended numerical vector), resolving the contradiction between semantic processing and numerical preservation.
2Adaptability or versatility
If traditional embedding vectors are used, then semantic similarity between words is captured, but numerical properties and ordering are not preserved
Solution Approach 1:
The patent merges two different types of information representation into a unified extended embedding vector: semantic information from word embeddings and numerical information from dedicated numerical vectors. This merging allows the model to simultaneously utilize both semantic similarity (for understanding word meaning) and numerical properties (for exact value comparison and ordering), thereby resolving the contradiction between adaptability and information preservation.
Solution Approach 2:
The extended embedding vector acts as a composite structure combining two distinct information types: the original embedding vector (providing semantic characteristics) and the appended numerical vector (providing numerical characteristics). This composite structure enables the model to leverage both semantic understanding and numerical precision, effectively resolving the contradiction between versatility and information loss.
3Ease of operation
If numerical tokens are treated as regular words, then they can be embedded, but the precise numerical values and relationships are lost
Solution Approach 1:
The patent applies local quality by treating numerical tokens differently from regular words specifically at the numerical vector component. While the embedding part treats all tokens uniformly (maintaining simplicity), the numerical vector part specifically preserves exact numerical values and relationships for numerical tokens. This localized differentiation resolves the contradiction by maintaining processing simplicity through uniform embedding while achieving numerical precision through specialized numerical vector representation.
Data Source
AI summary
An apparatus comprises processing circuitry configured to pre-process text data for inputting to a trained model, the pre-processing comprising: receiving a set of text data including numerical information, the set of text data comprising a plurality of tokens, wherein a first subset of the plurality of tokens comprises tokens that do not comprise numerical information, and a second subset of the plurality of tokens comprises tokens that each comprise respective numerical information; transforming each of the plurality of tokens into a respective encoding vector, each of the plurality of tokens in the second subset having a common encoding vector; assigning a respective numerical vector to each of the plurality of tokens, wherein each token in the second subset is assigned a respective numerical vector in dependence on the numerical information in said token; and combining the encoding vectors and numerical vectors to obtain a vector representation of the text data.