Switching Inference Neural Networks for Embedded Accuracy Stability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network inference methods face accuracy degradation due to differences in numerical formats between training and inference phases, particularly when transitioning from high-performance general-purpose computers to low-performance embedded devices, leading to reduced dynamic range and resolution.
Innovation Solution
An inference execution method that selects and switches among multiple inference neural network candidates with varying dynamic ranges and resolutions, adapting to environmental conditions and accuracy changes during the inference phase.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single neural network is used for inference on embedded devices, then device complexity is reduced, but accuracy degrades due to numerical format differences
Solution Approach 1:
The patent divides the neural network inference system into multiple candidate networks with different numerical formats (floating-point, fixed-point, integer) and precisions. Instead of using a single neural network, the system segments the inference task across multiple specialized networks, each optimized for specific numerical representations. This allows the system to maintain high accuracy across different device capabilities while managing complexity through structured organization of candidate networks.
Solution Approach 2:
The patent implements dynamic selection of neural network candidates based on environmental conditions, device capabilities, and accuracy requirements. The system dynamically switches between different candidate networks during runtime, adjusting the inference model to match current operational context. This dynamic adaptation resolves the contradiction by allowing the system to use simpler networks when appropriate while maintaining the option to use more complex, higher-precision networks when needed.
2Adaptability or versatility
If multiple inference neural network candidates are prepared, then accuracy is improved through dynamic selection, but device complexity increases
Solution Approach 1:
The patent performs preliminary preparation of multiple candidate neural networks with different numerical formats and precisions before runtime inference. During the inference phase, the system only needs to select from pre-prepared candidates rather than dynamically creating or converting networks. This preliminary action reduces runtime complexity while maintaining adaptability, as the selection process becomes a simpler choice among predetermined options optimized for different conditions.
Solution Approach 2:
The patent manages multiple candidate networks that differ by key parameters such as numerical format (floating-point, fixed-point, integer) and precision level. The selection mechanism changes these parameters based on environmental conditions and device capabilities, allowing the system to adapt to different scenarios. This parameter-based organization simplifies management compared to handling entirely different network architectures, as the core structure remains consistent while only numerical representation parameters vary.
3Measurement precision
If neural network is optimized for high-performance computers, then training accuracy is improved, but inference performance degrades on low-performance embedded devices
Solution Approach 1:
The patent segments the inference execution into multiple candidate networks optimized for different performance levels and device types. High-precision floating-point networks maintain training accuracy for complex tasks, while simplified fixed-point and integer networks provide fast execution on resource-constrained embedded devices. This segmentation allows the system to match network complexity with device capability, resolving the contradiction between training accuracy and inference speed.
Solution Approach 2:
The patent changes numerical representation parameters (floating-point to fixed-point to integer) and precision levels to match device capabilities. High-performance computers can utilize full-precision floating-point arithmetic for accurate training and complex inference, while low-performance embedded devices switch to lower-precision formats that enable faster execution. This parameter adaptation maintains acceptable accuracy while dramatically improving inference productivity on resource-constrained devices.
Data Source
AI summary
An inference execution method includes: selecting an inference neural network from among a plurality of inference neural network candidates generated from one training neural network that has been trained; sequentially obtaining data; sequentially executing, on the data sequentially obtained, inference using the inference neural network; sequentially outputting results of the inference sequentially executed; and selecting a new inference neural network from among the plurality of inference neural network candidates and switching the inference neural network to be used in the execution of the inference to the new inference neural network during an inference execution period in which the data is sequentially obtained, the inference is sequentially executed, and the results of the inference are sequentially output.