Multi-Dimensional Query Execution with Sequential Vectorization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional hardware and software approaches for business intelligence (BI) analysis, such as row-based databases and multi-dimensional online analytical processing (OLAP), fail to provide sufficient speed and scalability to handle large volumes of data and concurrent user requests in dynamically changing environments, particularly in ad hoc reporting and big data analysis.
Innovation Solution
A new aggregation engine leveraging modern CPU hardware capabilities through multi-threading and vectorization, utilizing Map-Reduce algorithms with just-in-time compilation and optimized sorting algorithms to perform parallel data aggregation tasks, enhancing performance and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional row-based databases and multi-dimensional OLAP approaches are used, then data storage and basic query processing is achieved, but processing speed and scalability for big data analysis and concurrent users is insufficient
Solution Approach 1:
The system divides the data into data chunks and partitions the data space into multiple dimensions (factual, dimensional, and result data spaces). This segmentation enables parallel processing of data across multiple threads and cores, significantly improving processing speed while maintaining manageable system complexity through modular architecture.
Solution Approach 2:
The patent introduces a multi-dimensional data space model with factual data space, dimensional data space, and result data space. This dimensional transformation allows the system to handle complex queries more efficiently by pre-computing and organizing data across multiple dimensions, thereby improving processing speed without proportionally increasing architectural complexity.
2Adaptability or versatility
If traditional hardware and software approaches are used, then basic BI analysis is performed, but scalability in terms of number of concurrent users and processing complexity is insufficient
Solution Approach 1:
The system implements dynamic resource allocation and adaptive query processing. The multi-threaded architecture allows the system to dynamically adjust the number of active threads and data chunk assignments based on the number of concurrent users and query complexity, enabling linear scalability while maintaining high throughput performance.
Solution Approach 2:
By segmenting data into manageable chunks and distributing them across multiple threads, the system achieves scalability in handling concurrent users. Each thread can independently process data chunks, allowing the system to scale throughput by simply adding more threads without increasing architectural complexity.
3Productivity
If sequential processing is used, then simple query execution is achieved, but performance for compute-intensive tasks and ad hoc reporting is insufficient
Solution Approach 1:
The query execution process is segmented into multiple independent threads that can process different data chunks simultaneously. This segmentation enables parallel execution of compute-intensive operations, achieving 10-15 times faster sorting and improved throughput while keeping each individual thread's complexity manageable.
Solution Approach 2:
The system performs preliminary data organization and pre-computation of dimensional data space structures before query execution. This preliminary action reduces the complexity of subsequent query processing by having data already organized in the multi-dimensional space, enabling faster query execution without requiring complex real-time processing algorithms.
Data Source
AI summary
A method is for executing a query in a multi-dimensional data space. The method includes performing operations with source data on an atomic level of the multi-dimensional data space, performing a mapping data operation on data from the operations, and performing a map consolidation operation on the mapped data from the operations. The method also includes performing a map-reduce operation on the mapped data from the operations, the map-reduce operation including sequential vectorized operations, and generating a result for the query based upon the sequential vectorized operations.