Integrated Data Warehouse and In-Memory Database for Live Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computing systems face delays and inefficiencies in data processing due to batch processing, lack of integration between data warehouses and in-memory databases, leading to latency, inconsistent data, and reduced analytical capabilities, which impact the timeliness and accuracy of insights and decision-making.
Innovation Solution
Integration of an analytic data warehouse with an in-memory database for batch and live streaming data processing, enabling efficient handling of large volumes, real-time insights, and advanced analytics by leveraging the strengths of both technologies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If batch processing is used in conventional systems, then data processing can handle large volumes, but processing delays occur between data collection and insight generation
Solution Approach 1:
The system segments data processing into two distinct pathways: batch processing for historical data storage and analysis in the data warehouse, and real-time streaming processing for immediate insights. This segmentation allows each pathway to optimize for its specific function, eliminating the trade-off between volume handling and speed.
Solution Approach 2:
An in-memory database serves as an intermediary layer between the data warehouse and streaming processing. It receives data from the warehouse, maintains it in memory for rapid access, and enables real-time analytics without the delays associated with traditional batch processing cycles.
2Device complexity
If data warehouse and in-memory database are not integrated, then system complexity is reduced, but data consistency and analytical capabilities deteriorate
Solution Approach 1:
The patent merges the data warehouse and in-memory database into a unified integrated system. The data warehouse provides structured storage and historical analysis, while the in-memory database provides real-time processing capabilities. This combination delivers consistent, reliable data across both batch and streaming operations.
Solution Approach 2:
The integrated system provides multi-functionality by enabling both traditional batch analytics and real-time streaming analytics through a single unified architecture. This universal system handles diverse analytical workloads without requiring separate independent systems.
3Device complexity
If traditional batch processing is used, then system simplicity is maintained, but responsiveness and real-time analytical capabilities are reduced
Solution Approach 1:
The system transitions from static batch processing to dynamic streaming processing. Data flows continuously through the system with real-time processing capabilities that adapt to changing analytical requirements, enabling the system to respond dynamically to new data as it arrives while maintaining architectural simplicity through the unified in-memory database interface.
Data Source
AI summary
A method for processing batch and live streaming data is disclosed. The method may include receiving a first data object from a data source, the first data object being of information technology event data; processing the first data object; determining, by utilizing a message queue, a first identifier for the first data object; storing the processed first data object in a data sink layer; transferring the first data object to a data warehouse configured to apply an analytic algorithm to the first data object, wherein the transfer is based on the first identifier; determining, by applying the analytic algorithm to the first data object, a third data object; transferring the first data object and third data object to an in-memory database; retrieving and presenting the first data object and/or third data object from the in-memory database to present in a presentation layer of a user interface.