Cumulative Incremental Feature Store for Online Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Online systems face challenges in updating feature values in real-time due to computation intensity, leading to delayed reflection of recent user behavior changes, which results in irrelevant actions and uninteresting information presentation.
Innovation Solution
Implementing a cumulative and incremental feature store system that combines partial results from both stores, with a decay factor to weight older actions, allowing for efficient updating and merging of feature values to reflect recent user interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If feature values are updated frequently to reflect recent user behavior changes, then the relevance of information presented to users improves, but the computational complexity and resource consumption increase significantly
Solution Approach 1:
The feature store is segmented into two distinct components: a cumulative feature store containing historical feature values and an incremental feature store containing recent feature value changes. This segmentation allows the system to query only the incremental portion for recent updates, significantly reducing computational complexity while maintaining reliability by combining both stores to get complete feature values.
2Productivity
If feature values are updated in real-time to reflect user actions, then user engagement improves, but the system response time and processing overhead increase
Solution Approach 1:
The system pre-computes and stores feature value changes in the incremental feature store as user actions occur, rather than computing complete feature values on-demand. This preliminary action captures feature changes incrementally as they happen, allowing fast retrieval when queries are made while maintaining real-time responsiveness and high user engagement.
3Loss of information
If the cumulative feature store is queried for all feature values, then complete historical context is obtained, but the query response time increases due to large data volume
Solution Approach 1:
The system extracts only the recently changed feature values into a separate incremental feature store, leaving the cumulative feature store with historical data. When a query is made, the system extracts and returns only the incremental portion first, then supplements it with historical context from the cumulative store only when needed, dramatically reducing query response time while preserving complete historical context.
Data Source
AI summary
Online systems, for example, social networking systems store features describing relations between entities represented in the online system. The information describing the features is represented as a graph. The online system maintains a cumulative feature graph and an incremental feature graph. Feature values based on recent user actions are stored in the incremental graph and feature values based on previous actions are stored in the cumulative graph. Periodically, the information stored in the incremental feature graph is merged with the information stored in the cumulative feature graph. The incremental graph is marked as inactive during the merge and information based on new user actions is stored in an active incremental feature graph. If a request for feature information is received, the feature information obtained from the cumulative feature graph, inactive incremental feature graph and the active incremental feature graph are combined to determine the feature information.