Common Fact Table for Distributed Data Warehouse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data mining systems face inefficiencies and inconsistencies when handling large data sets from usage logs, particularly due to the need for separate processing by each user and the challenges of handling unexpected dimension values, leading to slower response times and potential data inconsistencies.
Innovation Solution
The implementation of a common fact table with distributed storage of dimension keys, allowing multiple users to access aggregated measures and dimensions, and automatic discovery of dimension values using rule sets to validate and manage unexpected values, thereby enhancing data mining efficiency and consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If separate processing is performed for each user on large data sets from usage logs, then each user can have customized data analysis, but processing time increases and response times slow down
Solution Approach 1:
The patent pre-processes usage logs into a common fact table and generates auxiliary data structures before users need the data. This preliminary action stores aggregated measures and dimensions in an optimized format, so when users submit queries, the system can quickly retrieve pre-computed results rather than processing raw logs in real-time, thus maintaining customization while reducing processing time
Solution Approach 2:
The patent segments the large usage log data into a structured common fact table with separate auxiliary data structures for different dimensions (time, category, brand, etc.). This segmentation organizes data by measurement type and dimension, allowing users to access only the specific segments they need for their customized analysis, reducing the amount of data processing required while maintaining analysis flexibility
2Reliability
If multiple users access separate auxiliary data structures, then each user has dedicated data structures, but data inconsistencies may occur and storage efficiency decreases
Solution Approach 1:
The patent merges multiple users' auxiliary data structures into a single common fact table that serves all users. This common fact table contains aggregated measures and dimensions that are shared across all users, ensuring data consistency since everyone queries the same underlying data structure. The system maintains user-specific query capabilities while using shared storage, improving storage efficiency while preventing data inconsistencies
Solution Approach 2:
The common fact table is designed as a universal data structure that serves multiple users simultaneously. It contains dimension tables and auxiliary structures that can be queried by any user for their customized analysis needs. This multi-functional design allows the same data structure to support diverse user queries without requiring separate copies, thereby improving storage efficiency while maintaining data consistency across all users
3Measurement precision
If unexpected dimension values are manually validated and managed, then data accuracy is maintained, but human intervention increases and processing efficiency decreases
Solution Approach 1:
The patent implements self-service validation mechanisms where the system automatically detects, validates, and manages unexpected dimension values without human intervention. The auxiliary data structures include dimension tables that automatically check values against expected ranges and formats, and the system can autonomously handle outliers or invalid entries through predefined rules, maintaining data accuracy while eliminating manual validation steps and improving processing efficiency
Data Source
AI summary
Methods and data structures are provided for allowing data mining with improved efficiency. During processing of a usage log (or multiple logs) for an activity, such as a usage logfile of network search activity, a common fact table is generated. The common fact table allows a plurality of auxiliary data structures to be formed from the common fact table. These auxiliary data structures are designed to allow users to submit queries against the contents of the data structure in order to investigate the data. The efficiency of access of the common fact table is improved by allowing users to access auxiliary data structures other than the auxiliary data structures that are associated with a user. Optionally, the common fact table and/or the auxiliary data structures can include dimension values that correspond to both pre-identified dimension values as well as dimension values that are identified during processing of the activity logfiles.