Real-Time Feature Engineering for Low-Storage Online Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The excessive storage pressure and resource wastage caused by pre-storing large numbers of model entry features in traditional online prediction systems, leading to increased storage costs and reduced query timeliness.
Innovation Solution
A real-time prediction method that processes target original features on demand using feature engineering services to generate model entry features, reducing the need to store all model entry features in advance, and instead generating them dynamically based on user requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all original features and model entry features are pre-stored in the server database, then the demands of all users are met, but huge storage resources are occupied and query timeliness is significantly impacted
Solution Approach 1:
The patent pre-processes and stores only the necessary original features in the database before online prediction, rather than pre-storing all possible model entry features. This preliminary preparation of raw data eliminates the need to store derived features for all users, reducing storage consumption while ensuring that required features can be generated on-demand during online prediction to meet user demands.
2Reliability
If all original features and model entry features are pre-stored in the server database, then the demands of all users are met, but query timeliness is significantly impacted
Solution Approach 1:
The patent performs preliminary preparation by storing only original features in the database before online prediction. During online prediction, model entry features are generated on-demand from these pre-stored original features through feature engineering services. This approach eliminates the time-consuming query of pre-stored model entry features for all users, significantly improving query timeliness while still meeting user demands through rapid on-demand feature generation.
3Reliability
If model entry features are generated in advance for all users, then all user demands are met, but resource wastage occurs for small user groups
Solution Approach 1:
The patent implements a on-demand feature generation mechanism where model entry features are created only when actually needed during online prediction for specific users. The feature engineering services automatically generate the required model entry features from pre-stored original features at the moment of prediction, eliminating the waste of resources that would occur from pre-generating features for all users including small user groups. This self-service approach ensures full user demand coverage without resource wastage.
Data Source
AI summary
Provided are a real-time prediction method, an electronic device and a non-transitory computer-readable storage medium, which relate to the field of data processing, and in particular to the field of artificial intelligence. The specific implementation solution is acquiring a user request based on a business scenario; acquiring, from a plurality of pre-stored original features, at least one target original feature corresponding to the user identifier; determining, from a plurality of pre-trained models, a target model corresponding to the scenario identifier, and determining, from a plurality of feature engineering services, a target feature engineering service corresponding to the target model; performing real-time feature processing on the at least one target original feature by means of the target feature engineering service, so as to obtain a target model entry feature; and inputting the target model entry feature into the target model.