Deep Forest Decision Tree Layers for Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models, particularly neural networks, face inefficiencies in CPU environments due to the CPU's unsuitability for performing floating-point matrix operations, leading to slow inference speeds in resource-constrained scenarios.
Innovation Solution
Replace certain network layers of deep learning models with decision tree models, specifically deep forest models, which are trained to simulate the input and output of neural network layers, allowing CPUs to process these models more efficiently, leveraging their capability for logical operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If deep learning models use neural networks with floating-point matrix operations, then accuracy is maintained, but inference speed on CPUs becomes slow
Solution Approach 1:
The patent segments the neural network into individual layers and replaces each layer with a corresponding decision tree model. This segmentation allows the computationally intensive floating-point matrix operations to be divided into simpler logical operations that CPUs can process more efficiently, while maintaining the functional equivalence of each layer through the decision tree approximation.
Solution Approach 2:
The patent substitutes the mechanical floating-point matrix operation system with a logical decision tree system. By replacing the continuous arithmetic operations of neural networks with discrete decision tree structures, the system leverages CPU strengths in logical processing while avoiding the computational bottlenecks of floating-point operations.
2Productivity
If CPU is used for processing deep learning models, then resource constraints are satisfied, but processing speed decreases due to unsuitability for floating-point operations
Solution Approach 1:
The patent changes the fundamental parameter of computation from floating-point arithmetic to logical operations. By transforming the mathematical operations of neural networks into decision tree logic, the system optimizes for CPU architecture strengths, achieving faster processing speeds while reducing the computational complexity burden on the processor.
3Speed
If neural network layers are replaced with decision tree models, then inference speed increases on CPUs, but model accuracy may be compromised
Solution Approach 1:
The patent employs feedback mechanisms during the training and evaluation process to ensure that decision tree models maintain accuracy comparable to the original neural networks. By iteratively training and validating the decision tree layers against the neural network outputs, the system refines the approximations to preserve predictive performance while achieving faster inference.
Data Source
AI summary
Disclosed are techniques for modifying deep learning models (such as neural networks) to run more efficiently in computing environments with limited floating point computation resources. A deep learning model is trained using a set of training data. Input and output values are then recorded from the layers of the trained model when supplied with the training data, which are then used to generate deep forest decision tree models corresponding to individual layers of the trained model. Experimental versions of the trained model are then generated with different layers of the trained model replaced with their corresponding deep forest decision tree models. These experimental versions are then ranked according to the accuracy of their results compared to the results of the trained model. An updated trained model is then generated with one or more layers replaced with their corresponding deep forest decision tree models.