Decision-Tree Inference Thread Pool for Low-Latency Tree Traversal

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing decision-tree based ensemble methods face significant latency issues due to the time required to start multiple threads, which negatively impact inference performance in real-time applications with strict low-latency requirements, such as fraud detection and anti-money laundering operations.

Innovation Solution

Implementing multiple tree-traversal engines in separate CPU threads that remain active and use spinlocks for synchronization, allowing immediate processing of prediction requests and reducing latency by using atomic operations for synchronization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If multiple CPU threads are started for real-time inference, then inference performance and parallelism are improved, but thread startup time increases latency for individual transactions

Engineering Contradiction:

Improveinference performanceVSAvoidthread startup time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent pre-creates and maintains a pool of CPU threads in an active state before inference requests arrive. This preliminary action eliminates the need to start threads at the moment of request, thereby removing the startup latency penalty while preserving the ability to process multiple transactions in parallel when needed

Inventive Principle:

Principle #10Preliminary action

2Loss of time

If multiple CPU threads are maintained active for immediate processing, then latency for individual transactions is reduced, but system complexity and resource consumption increase

Engineering Contradiction:

Improveinference latencyVSAvoidthread management complexity

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

The patent introduces a thread pool manager as an intermediary component that handles the complexity of thread creation, maintenance, and coordination. This mediator abstracts the complex thread management operations from the inference logic, making the system easier to manage while maintaining active threads for low-latency processing

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If multiple CPU threads are used for parallel processing, then throughput for batch processing is improved, but synchronization overhead and contention increase

Engineering Contradiction:

Improvebatch processing throughputVSAvoidsynchronization overhead

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The patent segments the thread pool into dedicated worker threads that each handle specific tasks independently. This segmentation reduces synchronization contention by allowing threads to operate autonomously on different portions of the workload, thereby maintaining high batch processing throughput while minimizing the energy overhead associated with synchronization operations

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20250356242A1Advanced exploitation of multithreading to boost machine learning inference performance for decision-tree based ensembles

Publication Date: 2025.11.20 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method for performing machine learning decision-tree based inferences includes generating multiple CPU threads on an inference function server and determining if an inference request which requires a tree traversal operation has been received. If an inference request which requires the tree traversal operation has been received, then immediately executing the tree traversal operation. If an inference request which requires the tree traversal operation has not been received, then determining if a stop thread request has been received, wherein if a stop thread request has been received, then stopping the CPU thread.