Transaction Log Cleansing and Aggregation for Retail Data Modeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Economic modeling in retail environments faces challenges with raw transaction log data, which often contains erroneous and anomalous entries, leading to insufficient or inaccurate predictions due to insufficient data points or information loss during aggregation.
Innovation Solution
A system and method for cleansing and aggregating transaction log data by removing erroneous entries and compressing it over dimensions like store, product, and time, allowing for more efficient and accurate modeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is aggregated over multiple dimensions to increase data volume for modeling, then statistical robustness is improved, but information loss occurs during aggregation
Solution Approach 1:
The patent applies preliminary data cleansing before aggregation to remove erroneous and anomalous entries. This preliminary action ensures that only high-quality data is aggregated, preventing information loss from bad data while still achieving sufficient data volume for statistical robustness in the aggregated datasets used for economic modeling.
2Quantity of substance
If raw T-LOG data is used directly for modeling, then data volume is sufficient, but erroneous and anomalous entries reduce prediction accuracy
Solution Approach 1:
The patent extracts and removes erroneous and anomalous entries from the raw T-LOG data through a cleansing process before modeling. This extraction of bad data points maintains sufficient data volume for statistical robustness while eliminating sources of prediction error, thereby improving reliability of the economic models.
Solution Approach 2:
Data cleansing is performed as a preliminary step before economic modeling to remove erroneous and anomalous entries. This preliminary action ensures that the modeling process works with high-quality data, improving prediction accuracy while maintaining sufficient data volume through subsequent aggregation of cleaned records.
3Reliability
If data is aggregated to improve statistical robustness, then model reliability is improved, but data processing time increases
Solution Approach 1:
The patent performs data cleansing as a preliminary action before aggregation and modeling. By removing erroneous and anomalous entries early in the process, the subsequent aggregation operates on cleaner data, reducing the computational burden and processing time while still achieving the necessary statistical robustness for reliable economic models.
Data Source
AI summary
A computer-implemented method prepares data for modeling. The method comprises storing data from customer sales transactions in a database and retrieving a dataset of the data from the database. The dataset may include promotion and merchandizing entries. The method includes cleansing the dataset to remove erroneous and anomalous entries. Cleansing the dataset may include determining a threshold value from the dataset and determining whether a value of the dataset exceeds the threshold value, and determining an out-of-stock status for a product from the dataset. The method includes aggregating the dataset over a plurality of dimensions of the transactional space including store, product, and time dimensions, and analyzing the dataset following the cleansing and aggregating steps within a model to predict attributes of subsequent sales transactions.