Data Asset Placement Using Recommendation-Aware Latency Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data search systems in enterprise organizations face inefficiencies due to organizational silos, leading to redundant data searches and high latency, as they lack effective collaboration between business units and fail to leverage user relationships for relevant data asset recommendations.
Innovation Solution
Implementing a search and recommendation engine that utilizes user interaction data and self-supervised learning to provide personalized data asset suggestions based on past behavior, leveraging collaborative filtering and generative models to optimize data placement and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in centralized data centers, then data consolidation is achieved, but search latency increases due to distance from users
Solution Approach 1:
The patent implements local data caching at the edge network layer closer to users. Frequently accessed data assets are cached in local storage at edge locations, enabling faster access without requiring users to retrieve data from distant centralized data centers, thus reducing search latency while maintaining consolidation benefits
2Ease of operation
If keyword-based search is used, then simple data matching is achieved, but relevant data recommendations are insufficient
Solution Approach 1:
The patent implements a feedback mechanism where user interactions with data assets (views, downloads, searches) are logged and used to train machine learning models. These models continuously improve recommendation accuracy by learning from user behavior patterns, enabling the system to predict and recommend relevant data assets before users explicitly search for them
Solution Approach 2:
The system performs self-learning through unsupervised machine learning algorithms that automatically analyze user interaction patterns and data access trends. The system serves itself by autonomously improving its recommendation capabilities without requiring manual retraining or external intervention, adapting to changing user needs over time
3Loss of time
If data assets are placed close to users, then search latency is reduced, but data distribution complexity increases
Solution Approach 1:
The patent implements proactive data placement using predictive analytics to forecast which data assets users will need and pre-loads them into edge caches before actual requests occur. This preliminary action reduces latency by having data ready at the edge, while the system manages distribution complexity through automated intelligent placement algorithms
Data Source
AI summary
A search engine responding to a user query to find relevant data assets in a federation business data lake (FBDL) system based on interactions of known users interacting with data assets in the FBDL system. Data assets are optimally placed for minimal latency or maximal load. Data asset recommendations and past data asset access information are input as features to a time-series model for predicting future data access patterns. An expected latency and load risk is then determined and scored by a weighted mean of these values, and placement optimization is simulated using an optimization method (e.g., genetic algorithm). Using the scoring and simulation, a data asset placement engine is then used to move the locations of the data assets to minimize latency and/or to minimize maximal load.