Map-Reduce Data Cube Generation for Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Query execution on data cubes is time-consuming due to the need for rebuilding the cube for each query, and existing solutions require significant memory and are slow for complex queries from multiple cells, as results need to be computed dynamically.
Innovation Solution
A system and method using a map-reduce program in a data processing cluster to generate key-value pairs from source data, aggregate values, and store pre-computed query results in a distributed file system, allowing for fast response times and reducing memory requirements by caching pre-computed results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If pre-computing all or portions of the cells in a data cube is done, then query response time is improved, but memory requirements increase significantly
Solution Approach 1:
The patent segments the data cube computation by dividing it into multiple output files stored in a distributed file system. Each file contains results for specific queries or query patterns, allowing the system to pre-compute and store only necessary portions rather than the entire data cube in memory.
Solution Approach 2:
The patent transitions from in-memory storage to disk-based distributed storage, adding a new dimension of persistence and scalability. By storing pre-computed results in a distributed file system rather than volatile memory, the system achieves both fast response times and reduced memory requirements.
2Adaptability or versatility
If complex queries for data from multiple cells are executed dynamically, then query flexibility is maintained, but query execution speed decreases
Solution Approach 1:
The patent performs preliminary computation of query results and stores them in output files within a distributed file system. When queries are executed, the system retrieves pre-computed results from storage rather than computing them dynamically, significantly improving execution speed while maintaining query flexibility through the ability to handle various query types.
Data Source
AI summary
In accordance with an embodiment, described herein is a system and method for generating a data cube for analytics. A map-reduce program running in a data processing cluster can read each line of a source data, and generate a key-value pair for each of a plurality of data combinations in that line of data. Values paired with the same key can be aggregated to generate one or more frequency values or one or more aggregate values, for representing the result of a query on the source data. Each query result can be stored in an output file, and can be encapsulated into a data cube cached in a distributed file system of the data processing cluster. The data cube can map a query from a client application to an output file, and can return a pre-computed result in the output file from the data cube to the client application.