Hierarchical Token Encoding for Cross-Modal Attribute Fidelity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern machine learning systems face challenges in effectively representing non-textual data, particularly structured signals, due to their hierarchical nature and cross-modality adaptation issues, leading to inefficient training and costly inference.
Innovation Solution
A multi-modal AI system encodes data strings into tokenized strings using a text-based deep neural network (DNN), where each unit is represented by hierarchical tokens, allowing for both data string and hierarchy directionality, and employs iterative retraining to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If continuous or high-dimensional attributes of structured signals are discretized into a manageable vocabulary, then the vocabulary size is reduced to a practical scale, but the information fidelity is lost due to oversimplification
Solution Approach 1:
The patent segments continuous attributes into hierarchical discrete tokens organized by significance levels. Each attribute is divided into multiple tokens representing different precision levels (coarse-grained to fine-grained), allowing the model to select appropriate granularity for each prediction step, thus maintaining information fidelity while keeping vocabulary manageable.
Solution Approach 2:
The patent introduces a hierarchical dimension to the tokenization process, organizing tokens not just by vocabulary but by significance levels and precision grades. This multi-dimensional organization allows efficient navigation through attribute spaces without requiring a single massive vocabulary, resolving the contradiction between vocabulary size and information fidelity.
2Device complexity
If standard text-based models process sequences of discrete tokens, then the model architecture remains simple and efficient, but the hierarchical relationships within data units are not captured
Solution Approach 1:
The patent makes the token processing dynamic by allowing the model to adaptively select which hierarchical levels to process based on the prediction task. The hierarchical structure enables flexible navigation through different levels of detail, capturing relationships without requiring a fixed complex architecture for all scenarios.
Solution Approach 2:
The patent implements nested hierarchical tokens where coarser-grained tokens contain and organize finer-grained tokens. This nesting structure allows the model to process information at multiple levels simultaneously, capturing hierarchical relationships while maintaining a relatively simple base architecture that can be selectively activated.
3Productivity
If textual data dependencies are processed along a single sequential axis, then the processing is efficient and straightforward, but dependencies within the hierarchical breakdown of each unit's attributes are missed
Solution Approach 1:
The patent adds a hierarchical dimension to the sequential processing axis, creating a multi-dimensional processing space. The model can traverse both the sequential order of tokens and the hierarchical structure within tokens, capturing intra-unit dependencies without sacrificing the efficiency of sequential processing through attention mechanisms that operate across both dimensions.
4Adaptability or versatility
If pretrained knowledge in large-scale language models is transferred to non-textual data, then generalization improves, but cross-modality adaptation issues arise leading to suboptimal performance
Solution Approach 1:
The patent changes the parameter representation of non-textual data to match the expectations of text-based models through hierarchical tokenization. By transforming continuous attributes into discrete hierarchical tokens that mimic textual structure, the model can effectively transfer pretrained knowledge while adapting to non-textual domains, resolving the contradiction between generalization and accuracy.
Data Source
AI summary
A system is disclosed for encoding a data string of a first modality into a hierarchical tokenized representation for processing by a text-based deep neural network (DNN) trained on a second modality. The data string comprises multiple units, each having one or more attributes. Each attribute is represented in the tokenized string as a sequence of hierarchical tokens, with a first hierarchical token encoding one or more most significant bits and a subsequent hierarchical token encoding one or more less significant bits. The DNN processes the data string bidirectionally, across the sequence of units and within the token hierarchy, to select tokens that capture attribute information. The selected hierarchical tokens output by the DNN from a representation of the original data string that preserves attribute detail while enabling cross-modal processing using models trained on text.