In-Memory Insight Generation via Data Inference Engine
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional analysis tools require frequent manual intervention and are inefficient in generating ranked insights for large datasets with many measures and dimensions, leading to increased response times and maintenance efforts, and often provide outdated insights due to the lack of consideration for runtime inputs.
Innovation Solution
A Data Inference Engine provides statistically valid measure/dimension combinations to an analysis tool, which executes a master procedure at runtime, filtering and ranking insights without storing physical tables, and utilizing sub-procedures like outlier detection and percentage calculation to generate insights dynamically, reducing maintenance and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional analysis tools are used to generate insights for large datasets, then analytical insights can be provided, but frequent manual intervention is required and response time increases
Solution Approach 1:
The system pre-generates insights for all possible measure/dimension combinations before runtime, storing them in an in-memory database. This preliminary action eliminates the need for manual intervention and sequential processing during runtime, enabling instant retrieval and ranking of insights based on user queries.
Solution Approach 2:
The system dynamically ranks pre-generated insights at runtime based on user-specific criteria and query parameters. Instead of static pre-ranking, the system adapts the ranking order dynamically according to user needs, achieving both speed through pre-computation and relevance through dynamic adaptation.
2Adaptability or versatility
If pre-generated insights are provided by conventional analysis tools, then insights are available in advance, but they may not be relevant to runtime requests
Solution Approach 1:
All possible measure/dimension combination insights are pre-calculated and stored in memory before runtime. This comprehensive pre-computation ensures that no matter what user query arises, the relevant insight already exists and can be quickly retrieved and ranked, maintaining both relevance and validity.
Solution Approach 2:
The system uses user queries as feedback to dynamically rank pre-generated insights. The ranking mechanism incorporates user-specific criteria and query parameters, ensuring that the most relevant insights are presented first while maintaining statistical validity through the pre-computation phase.
3Productivity
If supporting intermediate physical tables are employed to achieve favorable performance, then query performance improves, but maintenance requirements increase
Solution Approach 1:
Instead of maintaining complex physical intermediate tables with all their durability, confidentiality, and integrity requirements, the system creates simplified in-memory copies of the necessary data structures. These in-memory representations provide the same query performance benefits without the overhead of persistent storage maintenance.
Solution Approach 2:
The system uses transient in-memory data structures that are created once and discarded after use, eliminating the need for long-term maintenance. These short-lived objects provide the necessary performance optimization during query execution without creating persistent maintenance burdens.
4Reliability
If sequential generation of insights is performed for each combination, then comprehensive coverage is achieved, but response time increases with more datasets and users
Solution Approach 1:
The system performs comprehensive insight generation for all measure/dimension combinations in advance, before any user queries are processed. This preliminary comprehensive generation ensures complete coverage of all possible insights, while the pre-computation eliminates sequential processing delays during runtime.
Solution Approach 2:
The system merges all individual insight generations into a single comprehensive pre-computation phase. Instead of sequentially generating insights for each user or query, all possible insights are generated once and stored, allowing multiple users to simultaneously access and rank insights without sequential delays.
Data Source
AI summary
Insights gleaned from statistical analysis of a dataset of an in-memory database, may be obtained with high throughput. In response to a user query, a Data Inference Engine (DIE) provides statistically valid measure/dimension combinations of the dataset to an analysis tool. A master procedure may be executed at search query runtime, allowing timely filtering of output of the analysis tool based on columns provided in the user request. Inputs/outputs of the master procedure may avoid storing physical tables, reducing maintenance. The master procedure may comprise individual sub-procedures (e.g. determining outliers, percentage contribution) executed in parallel. Output of sub-procedure(s) may be formatted per a final output format, facilitating union therebetween. Embodiments may filter certain measure/dimension combinations received from the DIE. For example, embodiments may consider only unique measure/dimension combinations (irrespective of dimension order), a limited number of combinations (e.g. ≦20), and/or combinations having a limited number of dimensions (e.g. ≦3).