Data Asset Placement Using Recommendations to Minimize Peak Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data search systems in enterprise organizations face inefficiencies due to organizational silos, leading to redundant data searches and manual efforts to identify relevant data assets, with inadequate collaboration between business units, resulting in wasted time and resources.
Innovation Solution
A search and recommendation engine that utilizes user interaction data and self-supervised learning to recommend data assets based on user behavior patterns, leveraging generative models for partial information scenarios, optimizing data placement and reducing latency by suggesting relevant data assets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in multiple data centers across different geographic locations, then data availability and redundancy are improved, but search latency increases due to greater distance from users
Solution Approach 1:
The patent segments data assets into different data centers based on user location and access patterns. By dividing the data storage infrastructure into geographically distributed segments, the system achieves both redundancy (through replication) and low latency (through local access), resolving the contradiction between reliability and speed.
Solution Approach 2:
The patent implements preliminary data placement decisions based on predictive models that forecast future data access patterns. By pre-positioning data assets in data centers likely to be accessed soon, the system reduces search latency while maintaining availability, addressing both reliability and speed requirements.
2Device complexity
If keyword-based search engines are used to find data assets, then implementation simplicity is maintained, but search effectiveness deteriorates due to inability to leverage user relationships and data asset relationships
Solution Approach 1:
The patent introduces a recommendation engine as an intermediary layer between users and the data asset search function. This mediator uses collaborative filtering and content-based filtering techniques to enhance search effectiveness by leveraging user relationships and data asset relationships, while maintaining the simplicity of the overall system architecture.
Solution Approach 2:
The patent implements feedback mechanisms where user interactions with data assets are continuously monitored and used to refine search results and recommendations. This feedback loop improves search effectiveness over time while the system maintains relatively simple implementation through automated learning processes.
3Device complexity
If users manually search through data assets without recommendations, then system complexity is minimized, but time consumption increases due to blind fishing through data
Solution Approach 1:
The patent implements self-service functionality where the recommendation engine automatically identifies and presents relevant data assets to users based on their profile and historical behavior. This eliminates the need for manual browsing and reduces search time while adding minimal complexity, as the system serves itself by automatically generating and updating recommendations.
Solution Approach 2:
The patent performs preliminary actions by pre-computing and storing user profiles and data asset metadata before actual search queries are executed. This preliminary processing enables rapid recommendation generation during user interactions, significantly reducing search time while maintaining acceptable system complexity through efficient data structures and algorithms.
Data Source
AI summary
A search engine responding to a user query to find relevant data assets in a federation business data lake (FBDL) system based on interactions of known users interacting with data assets in the FBDL system. Data assets are optimally placed for minimal latency or maximal load. Data asset recommendations and past data asset access information are input as features to a time-series model for predicting future data access patterns. An expected latency and load risk is then determined and scored by a weighted mean of these values, and placement optimization is simulated using an optimization method (e.g., genetic algorithm). Using the scoring and simulation, a data asset placement engine is then used to move the locations of the data assets to minimize maximal load that comprises a load risk representing how close a current load is to a service level agreement (SLA) requirement set by a system provider.