Hierarchical Tokenization for Cross-Modal AI Data Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern machine learning systems face challenges in effectively representing non-textual data, particularly structured signals with hierarchical attributes and cross-modality adaptation, leading to inefficient training and costly inference due to oversimplified information representation and suboptimal generalization.
Innovation Solution
A multi-modal AI system encodes data strings into tokenized strings using hierarchical tokens, where each unit is represented by most significant bits (MSBs) and less significant bits (LSBs), and employs a text-based deep neural network (DNN) to select these tokens based on both data string and hierarchy directions, trained using both text and non-text modalities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If continuous or high-dimensional attributes of structured signals are discretized into a manageable vocabulary, then training efficiency improves and inference cost reduces, but information fidelity deteriorates due to oversimplification
Solution Approach 1:
The patent segments continuous attributes into hierarchical discrete tokens organized by significance levels. Each attribute is divided into multiple tokens representing different precision levels, allowing the model to select appropriate granularity for each prediction task, thus balancing information retention with computational efficiency
Solution Approach 2:
The patent introduces a hierarchical dimension to the tokenization process, organizing tokens not just by vocabulary size but by significance levels and precision tiers. This multi-dimensional organization allows efficient navigation and selection of tokens based on task requirements, preserving information fidelity while maintaining manageable vocabulary sizes
2Device complexity
If standard text-based models process sequences of discrete tokens, then computational simplicity is maintained, but hierarchical relationships within data units are not captured
Solution Approach 1:
The patent implements nested hierarchical tokens where less significant bits are nested within the context of more significant bits. Each token level contains and references information from coarser levels, creating a nested structure that preserves hierarchical relationships while maintaining the sequential processing capability of standard text-based models
Solution Approach 2:
The patent makes the tokenization process dynamic by allowing the model to select different numbers and types of hierarchical tokens based on the prediction task requirements. The system can adaptively choose between using fewer coarse-grained tokens for simple tasks or more fine-grained tokens for complex tasks, optimizing both computational efficiency and information capture
3Reliability
If text-based DNN is pretrained on textual data, then language understanding capability is established, but cross-modality adaptation performance is suboptimal
Solution Approach 1:
The patent creates a universal tokenization framework that can represent multiple modalities (text, structured signals, tabular data) using the same hierarchical token system. This universal representation allows the pretrained text-based DNN to process diverse data types without requiring modality-specific architectures, significantly improving cross-modality adaptability while preserving language understanding capabilities
Solution Approach 2:
The patent changes the parameter representation from modality-specific continuous values to universal hierarchical discrete tokens. By transforming structured signal attributes into the same token format as text, the model can leverage its pretrained language understanding while adapting to new modalities through parameter transformation rather than architectural modification
Data Source
AI summary
A system is disclosed for encoding a data string of a first modality into a hierarchical tokenized representation for processing by a text-based deep neural network (DNN) trained on a second modality. The data string comprises multiple units, each having one or more attributes. Each attribute is represented in the tokenized string as a sequence of hierarchical tokens, with a first hierarchical token encoding one or more most significant bits and a subsequent hierarchical token encoding one or more less significant bits. The DNN processes the data string bidirectionally, across the sequence of units and within the token hierarchy, to select tokens that capture attribute information. The selected hierarchical tokens output by the DNN from a representation of the original data string that preserves attribute detail while enabling cross-modal processing using models trained on text.