Logistic Regression Gradient Calculation via Data Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training logistic regression models are limited by high memory requirements and slow training speeds due to the need for multiple threads to store full gradient vectors, making it difficult to handle large-scale features and models.
Innovation Solution
The proposed solution involves segmenting training data and weight vectors into smaller regions, allowing multiple threads to calculate sub-logistic regression gradients in parallel, which are then spliced to form a full gradient, reducing memory usage and increasing calculation speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple threads are started to calculate gradients in parallel, then calculation speed is improved, but memory consumption increases significantly
Solution Approach 1:
The patent segments the gradient calculation process by dividing the feature space into multiple partitions, where each thread calculates gradients for a specific partition. This allows parallel computation while each thread only stores a portion of the gradient vector, reducing total memory consumption from O(d) to O(d/N) where N is the number of threads and d is the feature dimension.
Solution Approach 2:
The patent introduces a new dimension of parallelism by partitioning the feature space across multiple threads rather than having each thread process all features. This dimensional division allows the system to scale computation across multiple threads while keeping per-thread memory usage constant, effectively decoupling calculation speed from memory consumption.
2Measurement precision
If more training data and larger features are used to depict long tails, then model accuracy is improved, but training difficulty increases
Solution Approach 1:
The patent segments the large-scale training data and feature space into manageable partitions that can be processed in parallel. This segmentation allows the system to handle super-large-scale logistic regression models with more training data and finer-grained features without overwhelming the computing resources, making training feasible while maintaining or improving accuracy.
Data Source
AI summary
The present disclosure provides logistic regression gradient calculation methods and apparatuses. One exemplary calculation method comprises: acquiring training data, the training data including X-row user data and Y-row click-through data corresponding to the X-row user data; converting the X-row user data into X-column data; segmenting the X-column data and a weight vector to form N X-column data segmentation blocks and N weight vector segmentation blocks; starting N threads respectively to generate N sub-logistic regression gradients according to the N X-column data segmentation blocks, the N weight vector segmentation blocks, and the corresponding Y-row click-through data; and splicing the N sub-logistic regression gradients to form a full logistic regression gradient. With embodiments of the present disclosure, a computing machine can support training of a super-large-scale logistic regression model, which increases the calculation speed, shortens the training time, and greatly reduces the memory usage of the computing machine.