Hierarchical Text Embedding for Noisy Conversation Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text analysis methods using phrase embeddings, such as word2vec, face challenges in capturing the semantic meaning of phrases due to arbitrary context window sizes and failure to recognize low-frequency multi-word terms, leading to noisy and unstructured representations in call center conversations.
Innovation Solution
A computer-based system and method for training and using text embedding models that create structured phrases from input text, using multiple levels of resolution (phrase, turn, and call) to generate input samples, and employing inverse document frequency measures for weighting terms, allowing for cleaner and more compressed embeddings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If a large context window size is used in word embedding training, then more context information is captured, but noisy neighbors are included which reduces embedding precision
Solution Approach 1:
The patent segments the context window into multiple hierarchical levels (word level, phrase level, sentence level, paragraph level, document level). Each level processes text with its own appropriate context window size, allowing precise local embeddings at the word level while capturing broader context at higher levels through separate processing stages.
Solution Approach 2:
The patent introduces a hierarchical dimension to the embedding process by creating embeddings at multiple levels of text organization. Instead of using a single flat context window, the system builds embeddings across five hierarchical dimensions, each contributing different aspects of context information with appropriate precision for that level.
2Ease of manufacture
If standard linear combination of word embeddings is used to create phrase embeddings, then the process is simple, but the semantic meaning of phrases is lost since phrase meaning is not the sum of constituent words
Solution Approach 1:
The patent segments phrase embedding creation into distinct hierarchical stages. At the phrase level, it explicitly preserves multi-word terms and their semantic relationships separate from individual word embeddings. This segmentation allows the system to maintain phrase-level semantics while still using simple linear combinations at the word level, resolving the contradiction between simplicity and semantic preservation.
Solution Approach 2:
The patent adds a phrase-level dimension to the embedding hierarchy. Instead of relying solely on word-level embeddings combined linearly, it creates a separate embedding layer for phrases that captures their intrinsic semantic meaning. This dimensional addition preserves phrase semantics while maintaining the computational simplicity of linear operations at each level.
3Ease of operation
If single-resolution sliding window training is used, then the training process is straightforward, but low-frequency multi-word terms are not recognized and representations become noisy
Solution Approach 1:
The patent segments the training process into multiple resolution levels corresponding to different text hierarchies. Each level uses sliding window training appropriate to its scale, with higher levels capturing broader context patterns. This segmentation enables reliable recognition of multi-word terms by training at the phrase level where they naturally occur, while maintaining operational simplicity through standardized training procedures at each level.
Solution Approach 2:
The patent introduces hierarchical resolution levels as an additional dimension to the training process. Instead of a single-resolution training approach, it implements training across five hierarchical dimensions, each contributing to the reliability of multi-word term recognition. The added dimensional complexity is offset by the modular structure that maintains operational simplicity at each level.
4Adaptability or versatility
If arbitrary context window sizes are used in embedding training, then flexibility is provided, but the precision of capturing semantic meaning deteriorates
Solution Approach 1:
The patent segments the context window flexibility into hierarchical levels, each with optimized window sizes appropriate to its scale. The word level uses small windows for precise local context, while higher levels use progressively larger windows to capture broader semantic patterns. This segmentation provides adaptability at each level while maintaining precision through level-appropriate parameter selection.
Solution Approach 2:
The patent applies local quality by optimizing context window parameters specifically for each hierarchical level rather than using uniform parameters throughout. Each level of the hierarchy has locally optimized settings that match its specific requirements, providing both flexibility and precision tailored to the local needs of each text organization level.
Data Source
AI summary
A system and method for training and using a text embedding model may include creating structured phrases from an input text; creating turn input samples from the input text, each turn input sample based on only or consisting of input from a single turn within the text and being formed by removing structure from structured phrases; and training an embedding model using the structured phrases and turn input samples. Call input samples may be created based on input from more than one turn within the text. At each level of resolution (e.g. phrase, speaker, call), a different level of resolution may be used to create input samples. At inference an embedding may be based on a weighted combination of the sub-terms within an input phrase, each weight being based on an inverse document frequency measure for the sub-term associated with the weight.