In-Memory Database Architecture for Big Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing architectures face challenges in rapidly evaluating large data sets due to storage requirements and complexity, leading to excessive processing times and resource utilization when analyzing transactions at different granularity levels.
Innovation Solution
A computing architecture that populates raw data into a normalized data model, transforms it into reporting optimized tables using a key figure data model, and loads these tables into an in-memory database for efficient data analysis, allowing for rapid evaluations with reduced resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in traditional physical storage systems, then storage capacity is sufficient for large data sets, but processing speed and evaluation time deteriorate
Solution Approach 1:
The patent segments data into two categories: frequently accessed data loaded into in-memory storage and less frequently accessed data remaining in physical storage. This segmentation allows rapid processing of active data while maintaining sufficient storage capacity for the complete data set, resolving the contradiction between fast evaluation speed and adequate storage capacity.
Solution Approach 2:
The patent introduces an in-memory database as an intermediary layer between physical storage and processing systems. This intermediary enables fast data evaluation by loading subsets of data into memory, while maintaining connection to the complete data set in physical storage, thus improving processing speed without sacrificing storage capacity.
2Productivity
If all raw data is loaded into memory for analysis, then processing speed improves, but memory resource consumption increases
Solution Approach 1:
The patent extracts only the necessary subsets of data from the complete data set and loads them into in-memory storage for analysis. By taking out only the required data portions rather than loading everything, the system achieves high productivity while consuming manageable memory resources.
Solution Approach 2:
The patent applies partial action by loading a subset of data into memory rather than the complete data set. This partial loading provides sufficient data for most analysis operations while avoiding the excessive memory consumption that would result from loading all data, thus improving productivity without proportionally increasing resource consumption.
3Ease of manufacture
If data is stored in normalized format, then data integrity and storage efficiency are maintained, but data analysis complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-processing and transforming data into analysis-ready formats during the data loading phase. Data is transformed from normalized storage format into denormalized, analysis-optimized structures when loaded into the in-memory database, eliminating the need for complex transformations during analysis operations.
Solution Approach 2:
The patent creates transformed copies of the normalized data in an in-memory database with optimized structures for analysis. These copies contain the same data integrity and relationships as the original normalized data but are organized in formats that simplify analysis operations, thus maintaining storage efficiency while reducing analysis complexity.
4Device complexity
If traditional data processing architectures are used, then system simplicity is maintained, but processing resource utilization increases
Solution Approach 1:
The patent segments the data processing architecture into distinct layers: physical storage for complete data sets, in-memory storage for active subsets, and processing layers for analysis operations. This segmentation improves resource utilization by ensuring that only necessary data is loaded into memory and processed, reducing overall processing resource consumption while maintaining manageable system complexity.
Data Source
AI summary
Raw data is received that characterizes each of a plurality of transactions. The data can be received, for example, via an adapter coupled to at least one data source. At least one first table is populated using the raw data according to a normalized data model. At least one second table is then populated, by applying a key figure data model to transform and enrich the at least one first table, such that it is optimized for data analysis operations. The at least one second table can include a subset of data stored in the at least one first table and additional enrichment data. At least one data analysis operation can later be initiated on the data populated in at least one second table to characterize one or more aspects of the transactions. Data can be provided that characterizes the initiated at least one data operation.