Text Token Compression and Embedding for Long-Context Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text processing methods, particularly those using large language models (LLMs) and neural networks, require significant computational resources and memory, leading to inefficient processing of long pieces of textual information.
Innovation Solution
A method involving parsing textual information into plaintext tokens, transforming them into binary tokens using lossless compression, and then embedding these tokens into vectorized form for processing by neural networks, followed by reverse transformations to obtain output tokens, thereby optimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large language models are used to process textual information, then the processing capability and accuracy are improved, but the computational resources and memory requirements increase significantly
Solution Approach 1:
The patent segments the textual information processing into distinct phases: binary transformation phase, embedding phase, neural network processing phase, and output phase. By dividing the text into tokens and processing them through separate transformation stages, the system reduces the memory footprint during each phase while maintaining overall processing accuracy.
Solution Approach 2:
The patent transforms the parameter representation of textual data from plaintext to binary format, then to embedding vectors. This parameter transformation reduces the memory requirements from storing large plaintext tokens to compact binary representations, and finally to optimized embedding vectors that preserve semantic meaning while requiring less computational resources.
2Reliability
If large language models process long pieces of textual information, then the completeness of analysis is improved, but the memory resources required increase
Solution Approach 1:
The patent extracts only the essential semantic information from long textual inputs by converting them to embedding representations. Instead of storing and processing the entire long text in memory, the system extracts key features into compact embedding vectors that capture the essential meaning, thereby maintaining analysis completeness while dramatically reducing memory requirements.
Solution Approach 2:
The patent transforms textual data from one dimension (plaintext tokens) to another dimension (embedding vectors in multi-dimensional space). This dimensional transformation allows the system to represent long texts in a compressed vector space where semantic relationships are preserved, enabling complete analysis of long documents with minimal memory usage.
3Productivity
If text is transformed through multiple stages (plaintext to binary to embedding), then the resource efficiency is improved, but the processing complexity increases
Solution Approach 1:
The patent performs preliminary binary transformation of text tokens before embedding. This preliminary action converts plaintext to binary format, which is then more efficiently processed by the embedding layer. By preparing the data in advance through binary conversion, the system optimizes subsequent processing steps and improves overall resource efficiency despite the additional transformation stage.
Data Source
AI summary
A method for processing a piece of textual information. The piece of textual information is parsed into a set of plaintext input tokens. Each of the plaintext input tokens is individually transformed using a first binary data transformation, to achieve a set of binary input tokens. Each of the set of binary input tokens is transformed individually or collectively, using an embedding data transformation, into one or several vectorized input tokens. The one or several vectorized input tokens is/are fed to a first neural network. A response is received from the first neural network in the form of one or several vectorized output tokens.