Parallel LDA in Shared-Nothing MPP Database
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing LDA implementations are not scalable or memory-efficient for large distributed shared-nothing massively parallel processing (MPP) databases, as they require significant memory to aggregate results and are not adaptable to relational databases, which are designed for interactive systems and parallel data access via SQL queries.
Innovation Solution
Implementing parallel LDA using Gibbs sampling within a shared-nothing MPP database by distributing documents across segment nodes, processing subsets locally, and using data structures like work tables to reduce memory consumption and improve efficiency through parallel processing and SQL-based operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If LDA results are aggregated in memory for processing, then processing speed is improved, but memory consumption increases significantly
Solution Approach 1:
The patent divides the corpus into multiple partitions distributed across different nodes in a shared-nothing architecture. Each node processes its local partition independently, maintaining only local topic models and counts rather than aggregating all data in a single memory space. This segmentation enables parallel processing while keeping memory consumption bounded per node.
Solution Approach 2:
The patent transitions from a single-node in-memory aggregation model to a distributed multi-node architecture where computation is distributed across spatial dimensions (multiple nodes) rather than concentrated in one location. This dimensional change allows the system to scale processing capacity without proportionally increasing memory requirements at any single node.
2Quantity of substance
If batch processing systems like MPI or Map/Reduce are used for LDA, then memory requirements can be met, but adaptability to relational databases is lost
Solution Approach 1:
The patent creates an LDA implementation that works within the relational database management system environment, combining the capabilities of batch processing with database operations. The system uses SQL queries for data access, leverages database transaction management, and integrates with existing database infrastructure, making it universally applicable to RDBMS platforms while maintaining LDA functionality.
Solution Approach 2:
The patent enables the database system itself to perform LDA computations using its existing resources and mechanisms. Rather than requiring external batch processing systems, the database uses its own query execution engine, memory management, and parallel processing capabilities to execute LDA algorithms directly, making the database self-sufficient for text analytics tasks.
3Productivity
If parallel processing is implemented in shared-nothing MPP databases, then scalability is improved, but memory efficiency deteriorates due to result aggregation requirements
Solution Approach 1:
The patent segments both the data (corpus partitioning) and the computation (distributed Gibbs sampling across nodes). Each node maintains only the topic models and sufficient statistics for its local data partition, avoiding the need to aggregate complete results in memory. This segmentation strategy enables linear scalability while maintaining constant memory usage per node regardless of total data size.
Solution Approach 2:
The patent introduces work tables as intermediary structures that store intermediate computation results on disk rather than in memory. These work tables serve as mediators between the parallel processing nodes and the final results, allowing nodes to write intermediate results to disk and read them back as needed, thereby avoiding large in-memory aggregations while maintaining parallel processing efficiency.
Data Source
AI summary
Latent Dirichlet allocation (LDA) analysis on a dataset is performed on an MPP relational database by distributing subsets of said dataset to a plurality of segments of the MPP database, and performing LDA analysis in parallel on the respective subsets on the plurality of segments using Gibbs sampling. An object library on each segment provides executable objects of user defined functions that can be called by an SQL query when the query requires functionality provided by an object.