In-Memory Aggregation Engine for Parallel BI Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional hardware concepts, such as Row-based databases and multidimensional online analytical processing (OLAP) approaches, fail to provide sufficient speed and scalability for business intelligence analysis, especially in handling big data and supporting a large number of concurrent users in a dynamically changing environment.
Innovation Solution
The implementation of an In-Memory Aggregation (IMA) system with a Massive Parallel Hardware accelerated aggregation engine, utilizing both CPU and GPU resources for massive parallel execution of compute-intensive tasks, combined with a columnar data storage and Analytical Query Language (AQL) optimized for general multidimensional data analysis, along with a modular processing system for scalable cloud computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional Row-based databases and multidimensional OLAP approaches are used, then data processing can be performed with standard hardware, but speed and scalability are insufficient for big data and concurrent users
Solution Approach 1:
The patent segments the monolithic OLAP processing into distributed microservices that can run independently across multiple hardware platforms. Each microservice handles specific analytical queries, allowing the system to scale horizontally by adding more service instances without increasing individual component complexity.
Solution Approach 2:
The patent creates a universal data processing platform that combines OLAP, data lake, and stream processing capabilities in a single system. The unified query engine can handle multiple data types and processing modes (batch, real-time, interactive) through a common architecture, eliminating the need for separate specialized systems.
2Productivity
If data is stored in traditional relational database format, then data integrity and ACID properties are maintained, but query performance for multidimensional analysis is insufficient
Solution Approach 1:
The patent implements local quality by storing data in columnar format optimized for analytical queries while maintaining relational integrity constraints. Different storage formats (columnar, row-based, compressed) are applied to different data subsets based on their access patterns, with hot data using high-performance formats and cold data using compressed storage.
Solution Approach 2:
The patent introduces an intermediary layer (unified query engine) that sits between the data storage and analysis tools. This engine handles query optimization, data format conversion, and integrity constraint enforcement, allowing fast columnar storage to maintain relational data integrity without requiring changes to the underlying storage structure.
3Adaptability or versatility
If hardware resources are increased to handle more concurrent users, then scalability improves, but cost and system complexity increase
Solution Approach 1:
The patent implements dynamic resource allocation where computing resources are allocated based on real-time query demands. The system can automatically scale resource allocation up or down, and can dynamically load-balance queries across available hardware resources without requiring a fixed over-provisioned architecture.
Solution Approach 2:
The patent enables self-service through automated query optimization and resource management. The unified query engine automatically optimizes query execution plans, selects appropriate data formats, and manages resource allocation without requiring manual intervention or complex configuration, reducing the operational complexity of scaling the system.
4Productivity
If in-memory processing is used to accelerate aggregation, then processing speed improves, but memory requirements and hardware cost increase
Solution Approach 1:
The patent applies partial in-memory processing where only the most frequently accessed data subsets (hot data) are kept in memory for fast aggregation, while less frequently accessed data remains on disk. The system dynamically determines which data to cache in memory based on query patterns, achieving speed improvements for critical queries without requiring full in-memory storage of all data.
Data Source
AI summary
An in-memory aggregation (IMA) system having a massive parallel hardware (HW) accelerated aggregation engine uses the IMA for providing maximal utilization of HW resources of any processing unit (PU), such as a central processing unit (CPU), general purpose GPU, special coprocessors, and like subsystems. The PU accelerates business intelligence (BI) application performance by massive parallel execution of compute-intensive tasks of in-memory data aggregation processes.