Nearline Personalized Model Updates via Modular Event Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems face challenges in handling and processing large, complex data sets, particularly in big data analytics, due to difficulties in collecting, storing, managing, and visualizing petabytes or exabytes of data generated continuously from multiple sources, leading to inefficiencies in data management and processing.
Innovation Solution
A system for nearline updates to personalized statistical models and features, which includes an event-processing apparatus, feature-processing apparatus, and model-training apparatus, that aggregates user responses to generate standardized events, update count and likelihood features, and retrain statistical models in real-time, allowing for timely reflection of user preferences in recommendations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional software tools and storage mechanisms are used to handle big data, then data collection and storage are simplified, but the system becomes unable to handle petabytes or exabytes of data in a timely manner
Solution Approach 1:
The system divides the data processing workload across multiple servers and nodes, with each component handling specific portions of the data. The event-processing apparatus, feature-processing apparatus, and model-training apparatus operate as separate modular components that can independently process data streams, enabling scalable handling of petabytes or exabytes of data without requiring a monolithic complex system.
Solution Approach 2:
The patent introduces intermediary components such as the feature-processing apparatus that acts as a mediator between event-processing and model-training. These intermediary layers process and prepare data in standardized formats, enabling efficient data flow between components and reducing the complexity of direct point-to-point connections between all system elements.
2Productivity
If massively parallel software is used to handle big data, then data processing capacity increases, but synchronization among servers and nodes becomes more difficult
Solution Approach 1:
The system performs preliminary processing of events into standardized formats by the feature-processing apparatus before data is distributed to model-training components. This preliminary standardization ensures that all subsequent processing operations receive data in a consistent format, eliminating synchronization issues that would otherwise arise from heterogeneous data formats across parallel servers and nodes.
3Measurement precision
If real-time model updates are implemented, then recommendation relevance to current user preferences improves, but system latency increases
Solution Approach 1:
The system dynamically adjusts the frequency and timing of model updates based on the nature of incoming events and user interactions. The model-training apparatus can operate in different modes - performing continuous incremental updates for high-relevance events while maintaining a baseline model for general recommendations. This dynamic approach enables near real-time updates for critical recommendations without the latency penalties of constant full model retraining.
Data Source
AI summary
The disclosed embodiments provide a system for processing data. During operation, the system obtains events reflecting responses by a user to job recommendations outputted to the user. Next, the system updates a set of features for the user from the events. The system then includes the updated set of features in a feature repository for use by a statistical model in generating a ranking of jobs for the user. Finally, the system retrains the statistical model using the events prior to using the statistical model to update the outputted job recommendations using the ranking.