Insight Store System for Big Data Analytics Reuse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for generating insights from big data are time-consuming due to extensive I/O operations during ETL processes and require manual effort for feature vector design and value estimation, also lacking efficient mechanisms for reusing existing insights.
Innovation Solution
An 'Insight store' system that processes data using analytics functions, associates metadata with insights, and stores them for immediate availability, enabling out-of-box analytics without platform setup, allowing insights to be generated and reused timely.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If ETL processes are used to process big data, then data can be transformed and loaded, but the process is time-consuming due to extensive I/O operations
Solution Approach 1:
The patent applies preliminary action by pre-processing and caching data in memory before it is needed for analytics. The system performs data preparation steps in advance, storing processed data in an in-memory cache, so that when analytics queries are executed, the data is already available without requiring time-consuming I/O operations during the actual analytics process.
2Adaptability or versatility
If manual feature vector design is performed, then analytics can be customized, but it requires significant manual effort and time
Solution Approach 1:
The patent applies self-service by enabling the system to automatically generate feature vectors and perform analytics without requiring manual intervention. The analytics engine automatically discovers patterns, generates relevant features, and executes queries against the cached data, eliminating the need for data scientists to manually design feature vectors while still providing customized analytics results.
3Productivity
If insights are generated and stored, then business value can be derived, but there is no efficient mechanism to reuse existing insights in other workflows
Solution Approach 1:
The patent applies universality by creating an in-memory insight store that serves multiple analytics workflows simultaneously. Once insights are generated and cached in memory, they become universally accessible resources that can be reused across different analytics queries and workflows without regenerating them, making the insight generation capability adaptable and versatile across the entire analytics platform.
Data Source
AI summary
Some examples relate to associating an insight with data. In an example, data may be received. A determination may be made that data type of the data is same as compared to an earlier data. An insight generated from the earlier data may be identified, wherein the insight may represent intermediate or resultant data generated upon processing of the earlier data by an analytics function, and wherein during generation metadata is associated with the insight. An analytics function used for generating the insight may be identified.