Recommendation Engine Combining Real-Time and Batch Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing recommendation systems face challenges in handling large volumes of data, generating personalized recommendations, and failing to consider user-specific scenarios, particularly for anonymous users, with limited data processing capabilities and lack of transparency in backend processes.
Innovation Solution
A system and method that combines real-time and batch data processing using a distributed database and multiple machine learning techniques to generate personalized recommendations, allowing for customization and transparency through a processor-based recommendation engine.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If existing recommendation systems use single analytical technique, then system complexity is reduced, but recommendation accuracy and personalization deteriorate
Solution Approach 1:
The patent combines multiple analytical techniques (collaborative filtering, content-based filtering, contextual bandits, reinforcement learning) into a unified recommendation system that processes different data types and scenarios simultaneously, achieving superior personalization accuracy while managing complexity through modular architecture
Solution Approach 2:
The system uses a composite analytical approach by integrating diverse machine learning models and analytical techniques that work together synergistically, similar to how composite materials combine different substances to achieve properties superior to individual components
2Device complexity
If existing recommendation systems process data in batch mode, then processing simplicity is maintained, but responsiveness to recent events deteriorates
Solution Approach 1:
The system performs preliminary processing of batch data to create pre-computed features and user profiles, then uses these pre-prepared structures to rapidly respond to real-time events without requiring complex real-time processing for every query
Solution Approach 2:
The patent divides data processing into distinct segments: batch processing for historical data and real-time processing for recent events, allowing each segment to be optimized independently for its specific requirements
3Device complexity
If existing recommendation systems use limited data sources, then data processing simplicity is improved, but personalization capability deteriorates
Solution Approach 1:
The system is designed to universally process multiple data types (explicit feedback, implicit feedback, contextual information) through a unified architecture that adapts to different data sources and scenarios, enabling comprehensive personalization without proportionally increasing processing complexity
4Device complexity
If existing recommendation systems act as black box, then system simplicity is maintained, but user transparency and control deteriorate
Solution Approach 1:
The system implements feedback mechanisms that provide users with visibility into recommendation generation processes, allowing users to understand why certain recommendations are made and to provide feedback that influences future recommendations, thereby reducing information loss and improving transparency
Data Source
AI summary
Disclosed is method and system for generating recommendations to a user. System receives real time data associated with users for scenarios and batch data associated with multiple users from different data sources, received from different data channels. The user is online user. System pre-processes batch data and real time data to generate pre-processed data and stores preprocessed data in distributed, scalable big data store. System filters pre-processed data based on rules to obtain filtered data. System applies combination of machine learning techniques on filtered data, based on the scenarios associated with the user, leveraging inter-play between machine learning techniques, to generate personalized recommendations for individual user and storing the personalized recommendations in distributed database. Machine learning techniques are customized to work in distributed processing mode to reduce overall processing time. System recommends user with the personalized recommendations comprising products or services.