Multi-Modal DNN Weight Calculation for Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning technologies face challenges in accurately and efficiently processing multi-modal data inputs, such as images, texts, and sounds, as they lack a method to effectively calculate and apply weights reflecting the importance of each data type for improved performance in specific tasks.
Innovation Solution
An electronic apparatus and method utilizing a deep neural network (DNN) that receives and processes input data of different types by extracting sub-feature information from pre-trained sub-networks, encoding it, and calculating weights for each type based on feature information from multiple layers, allowing for a final output value to be generated by applying these weights, thereby enhancing task performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning networks process multi-modal data inputs without calculating weights for each data type, then the processing speed is maintained, but the accuracy and performance for specific tasks deteriorate due to inability to reflect importance of each data type
Solution Approach 1:
The patent segments the weight calculation process by introducing separate weight calculation units for each data type (first weight calculation unit for first type data, second weight calculation unit for second type data). Each unit independently calculates weights for its respective data type based on feature information from the deep neural network, allowing parallel computation that maintains processing speed while improving accuracy through type-specific weight optimization.
Solution Approach 2:
The patent performs preliminary weight calculation by computing weights for each data type before the final data processing stage. The weight calculation units compute importance weights in advance based on feature information extracted from the deep neural network, so that when multi-modal data is processed, the already-calculated weights can be directly applied without adding computational overhead during the main processing flow.
2Productivity
If deep learning networks use uniform processing for all data types, then the device complexity is low, but the performance for specific tasks deteriorates due to inability to reflect importance of each data type
Solution Approach 1:
The patent applies local quality by making the processing characteristics specific to each data type. Different weight calculation units are dedicated to different data types (first weight calculation unit for first type, second for second type), allowing each unit to optimize weights locally for its specific data type's characteristics and importance to the task, thereby improving overall performance without requiring complete redesign of the entire system.
Solution Approach 2:
The patent achieves universality through the deep neural network structure that processes multiple data types (first type and second type data) through a unified architecture. The network extracts feature information from all data types and feeds it to multiple weight calculation units, allowing the same neural network backbone to serve multiple functions while type-specific weight calculations adapt the processing to each data type's needs.
3Measurement precision
If weights for each data type are calculated based on feature information from multiple layers, then the accuracy improves by reflecting importance of each mode, but the computational time increases
Solution Approach 1:
The patent performs preliminary weight calculation by computing weights for each data type in advance based on feature information extracted from the deep neural network. The weight calculation units compute importance weights before the final data processing stage, so that when multi-modal data is processed, the already-calculated weights can be directly applied without adding computational overhead during the main processing flow.
Solution Approach 2:
The patent segments the weight calculation process by introducing separate weight calculation units for each data type. Each unit independently calculates weights for its respective data type based on feature information from the deep neural network, allowing parallel computation that reduces the total computational time compared to sequential processing, while maintaining accurate type-specific weight optimization.
Data Source
AI summary
An electronic apparatus for performing a preset task by using a deep neural network (DNN), the electronic apparatus includes an input interface configured to receive input data of a first type and input data of a second type; and a processor configured to obtain first sub-feature information corresponding to the input data of the first type and second sub-feature information corresponding to the input data of the second type; obtain feature information from each of a plurality of layers of the DNN by inputting the first sub-feature information and the second sub-feature information into the DNN; calculate a weight for each type corresponding to each of the plurality of layers, based on the feature information, the first sub-feature information, and the second sub-feature information; and obtain a final output value corresponding to the preset task by applying the weight for each type, in each of the plurality of layers.