Fuzzy Token Pruning for Accurate Lightweight Document Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional token pruning methods for deep-learning-based document classification reduce model accuracy due to information loss and lack of consideration for individual token importance, necessitating a lightweight yet accurate document classification solution.
Innovation Solution
A deep-learning-based document classification method that utilizes fuzzy theory to calculate token importance scores, prune tokens based on these scores, and combine them with seed tokens to maintain accuracy while reducing model size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional token pruning is applied to lightweight the model, then model size and resource consumption are reduced, but classification accuracy deteriorates
Solution Approach 1:
The patent changes the pruning parameter from simple token frequency to token importance scores derived from attention weights. This parameter transformation allows selective retention of semantically important tokens while removing redundant ones, thereby maintaining classification accuracy while achieving model lightweighting
Solution Approach 2:
The patent applies different treatment to different tokens based on their local importance characteristics. Tokens are selectively pruned or retained based on their individual importance scores rather than uniform pruning, preserving critical information while removing redundancy
2Device complexity
If pruning is performed on the entire sequence, then model complexity is reduced, but information loss occurs
Solution Approach 1:
The patent transforms the pruning decision parameter from uniform frequency-based metrics to position-aware importance scores. This allows the model to identify and retain information-critical tokens while pruning less important ones, reducing complexity without significant information loss
Solution Approach 2:
The patent uses attention weights as feedback signals to guide the pruning process. Tokens with high attention weights (indicating high information value) are retained, while those with low weights are pruned, creating a feedback-driven selective pruning mechanism
3Ease of operation
If pruning is performed based on token frequency, then processing is simplified, but individual token importance is not considered
Solution Approach 1:
The patent changes the assessment parameter from token frequency to token importance scores based on attention weights. This provides a more precise measurement of token importance that reflects actual semantic contribution to classification, rather than just occurrence frequency
Data Source
AI summary
A document classification method is performed in a computing device equipped with one or more processors and a memory storing one or more programs executed by the one or more processors. The document classification method includes acquiring sequence data obtained by digitizing a document, and embedding the acquired sequence data to produce embedded data including one or more tokens, generating fuzzy-pruned data pruned by performing token pruning the embedded data, which is a set of tokens, based on fuzzy theory, producing combined tokens by combining tokens of the pruned fuzzy-pruned data with seed tokens of seed data, and generating combined data as a set of the combined tokens, and producing one classification value based on the combined data.