Sales Prediction Data Processing Using Difference Data for Imbalanced Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing methods struggle to accurately predict rare positive labels in imbalanced datasets, such as purchase or sales behavior, due to the difficulty in incorporating behavior-specific and merchandise-specific factors into statistical models.
Innovation Solution
A data processing apparatus that acquires and divides data based on external condition identification information and label information to generate difference data, which is then used for statistical analysis to calculate a regression coefficient vector that maximizes the AUC value on a ROC curve, effectively balancing the prediction of positive and negative labels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If statistical models are derived using accumulated data with imbalanced labels, then the model can be trained on available data, but the rare positive label cannot be accurately predicted
Solution Approach 1:
The patent introduces difference data as an intermediary representation that transforms the imbalanced classification problem into a regression problem. By computing differences between positive and negative label samples, the method creates a new feature space where the rare positive events can be accurately predicted without being constrained by their original scarcity in the dataset.
Solution Approach 2:
The patent changes the problem parameters by transforming the classification task into a regression task. Instead of directly predicting class labels, the model predicts the difference in feature values between positive and negative samples, then uses this difference to calculate prediction scores. This parameter transformation allows the model to learn from imbalanced data more effectively.
2Measurement precision
If factors such as behavior peculiar to a person who buys and sells and factors specific to merchandise are not included in the accumulated data set, then the data collection process remains simple, but these factors cannot be incorporated into the statistical model
Solution Approach 1:
The patent extracts the influence of behavior-specific and merchandise-specific factors by computing differences between samples. Instead of explicitly collecting and modeling these difficult-to-measure factors, the method extracts their net effect through difference calculations, which cancel out common variations and isolate the factors of interest.
Solution Approach 2:
The patent creates a transformed copy of the original data in the form of difference data. This copied representation preserves the essential information about behavior and merchandise factors while removing the confounding effects of class imbalance, allowing the model to learn from this simplified representation.
3Productivity
If the number of positive label samples is much smaller than negative label samples, then the data storage and processing remains efficient, but the rare positive label cannot be accurately predicted
Solution Approach 1:
The patent segments the learning process into two distinct stages: first computing difference data from pairs of positive and negative samples, then using this difference data for regression-based score calculation. This segmentation allows the model to efficiently process the imbalanced data while maintaining high prediction accuracy for rare positive events.
Data Source
AI summary
A data processing method according to an embodiment acquires data including a plurality of records, divides the data based on external condition identification information such as a user ID to generate data sets Di for respective external conditions, divides each of the data sets Di based on label information indicating whether the record corresponds to a positive label indicating that a predetermined event has occurred or a negative label indicating that the predetermined event has not occurred to generate two data sets Di+ and Di− for the respective label information, generates difference data for a combination of a record included in one data set of the two data sets and a record included in the other data set, combines the generated difference data to generate integrated data Dnew, performs statistical analysis using Dnew, and outputs a result of performing the statistical analysis.