Path-Based MIP Decision Tree Learning with Column Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing decision tree learning methods are NP-hard and rely on greedy heuristic-based methods, making it challenging to incorporate constraints, especially with sample-level constraints and linear metrics, and they do not scale well due to the high number of binary variables and constraints.
Innovation Solution
A path-based mixed-integer program (MIP) formulation is used to learn optimal decision trees, which splits the decision tree into multiway decision trees, allowing for the incorporation of intra-rule and inter-rule constraints, monotonic prediction output, and fairness constraints, and uses column generation to provide a scalable solution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If greedy heuristic-based methods are used for decision tree learning, then the algorithm is computationally efficient, but it is challenging to incorporate constraints and does not scale well
Solution Approach 1:
The patent segments the decision tree learning problem into multiple subproblems by decomposing the tree structure into individual paths from root to leaf. Each path represents a candidate decision rule, allowing constraints to be incorporated at the path level while maintaining computational tractability through column generation techniques.
Solution Approach 2:
The patent introduces column generation as an intermediary mechanism between the optimization objective and the constraint satisfaction. This intermediary allows the model to handle complex constraints (including sample-level constraints and linear metrics) by generating candidate paths on-demand and selecting the optimal subset, rather than directly solving the full constrained optimization problem.
2Adaptability or versatility
If MIP methods with arc-based formulation are used, then constraints can be handled, but the number of binary variables and constraints is high导致 poor scaling
Solution Approach 1:
The patent segments the arc-based MIP formulation into path-based formulation, where instead of modeling individual arcs and their constraints, the model directly represents complete paths from root to leaf as decision variables. This segmentation reduces the number of variables and constraints by eliminating redundant arc-level representations.
Solution Approach 2:
The patent transitions from arc-based formulation (lower dimensional representation of tree edges) to path-based formulation (higher dimensional representation of complete decision rules). This dimensional change allows the model to work with fewer variables by representing entire decision paths as single entities rather than sequences of arcs.
3Productivity
If path-based MIP formulation is used, then scalability is improved with column generation, but the formulation is more complex
Solution Approach 1:
The patent applies preliminary action by pre-generating a subset of candidate paths (columns) before solving the optimization problem. The column generation process starts with a restricted master problem using a preliminary set of paths, then iteratively adds new paths that improve the objective function, avoiding the need to enumerate all possible paths upfront.
Solution Approach 2:
The patent introduces dynamics into the formulation by making the set of decision variables (paths) dynamic rather than fixed. The column generation algorithm dynamically adds or removes paths from the master problem based on their contribution to the objective function, allowing the model to adapt the problem size during optimization rather than being constrained by a static variable set.
Data Source
AI summary
A computer-implemented machine learning method includes accessing a decision tree associated with a path-based machine learning model. The decision tree is split into a plurality of multiway decision trees in a path-based formulation, each of the plurality of decision trees having an attribute not occurring more than once in each of the plurality of decision trees. A problem associated with the machine learning model is solved using one or more of the plurality of decision trees in which one or more decision rules of the decision tree are mapped using a mixed-integer program (MIPS).