Distributed LDA Topic Modeling Using Approximate Counters

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing topic modeling algorithms face challenges in efficiently performing distributed processing due to high memory and communication bandwidth requirements, particularly in GPU-based systems, where inter-device communication slows down the processing speed.

Innovation Solution

A data-parallel algorithm for topic modeling using approximate counters, such as binary Morris, general Morris, and Csűrös approximate counters, is implemented to reduce memory usage and communication bandwidth, allowing for faster processing by aggregating and sharing 8-bit or 16-bit counter values across nodes in a distributed system.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional counter-based algorithms are used for topic modeling, then measurement precision is maintained, but memory usage and communication bandwidth requirements increase

Engineering Contradiction:

Improvecounter accuracyVSAvoidmemory usage

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The patent changes the parameter representation from conventional counters to approximate counters (e.g., binary Morris counters), which use fewer bits (e.g., 8-bit or 16-bit) to represent count values. This parameter change reduces memory usage and communication bandwidth while maintaining sufficient accuracy for topic modeling applications.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent employs approximate counters that can be easily created, transmitted, and discarded between distributed computing nodes. These lightweight counter representations enable frequent communication and aggregation operations without significant memory overhead, facilitating efficient distributed processing.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

2Measurement precision

If conventional counter-based algorithms are used for topic modeling, then measurement precision is maintained, but processing speed decreases

Engineering Contradiction:

Improvecounter accuracyVSAvoidprocessing speed

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

By changing from conventional counters to approximate counters with compact representations, the patent enables faster transmission and processing of counter values across distributed nodes. The reduced data size allows for quicker aggregation operations, improving overall processing speed while maintaining adequate precision for topic modeling.

Inventive Principle:

Principle #35Parameter changes

3Power

If distributed system architecture is used for topic modeling, then processing power is increased, but communication overhead increases

Engineering Contradiction:

Improvecomputing powerVSAvoidcommunication bandwidth

Core Design Contradiction:

PowerVSLoss of energy

Solution Approach 1:

The patent extracts only the essential counter value information needed for topic modeling and transmits this condensed data between distributed nodes. By using approximate counters that represent only the necessary statistical information, the system minimizes communication bandwidth consumption while maintaining distributed computing power.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent changes the parameter representation to approximate counters that require fewer bits for transmission. This parameter optimization reduces the communication overhead between distributed nodes, allowing the system to leverage multiple computing resources without proportionally increasing communication costs.

Inventive Principle:

Principle #35Parameter changes

4Productivity

If distributed system architecture is used for topic modeling, then processing speed is increased, but inter-device communication becomes a bottleneck

Engineering Contradiction:

Improveprocessing speedVSAvoidinter-device communication speed

Core Design Contradiction:

ProductivityVSSpeed

Solution Approach 1:

The patent uses lightweight approximate counter representations that can be quickly created, transmitted, and aggregated between distributed nodes. These compact counter objects minimize the time and bandwidth required for inter-device communication, reducing the bottleneck effect while maintaining high processing speeds through parallel computation.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS10140281B2Method and system for distributed latent dirichlet allocation computation using addition of approximate counters

Publication Date: 2018.11.27 ORACLE INT CORP

AI summary

Herein is described a data-parallel algorithm for topic modeling on a distributed system in which memory and communication bandwidth requirements are streamlined for distributed implementation. According to embodiments, a distributed LDA Gibbs sampling algorithm shares approximate counter values amongst the nodes of a distributed system. These approximate counter values are repeatedly aggregated and then shared again to perform the distributed LDA Gibbs sampling. In order to maintain the shared counter values as approximate counter values of sixteen bits or less, approximate counter values are summed to produce aggregate approximate counter values. These small aggregate approximate counter values are shared between the nodes of the distributed system. As such, the addition of various types of approximate counters is described herein. Specifically, addition of binary Morris approximate counters, general Morris approximate counters, and Csűrös approximate counters are described in the context of distributed implementations of an LDA Gibbs sampling algorithm.