Hierarchical Data Layering for Query Response Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Business intelligence systems face increased computing resource demands and longer query processing times as they handle larger volumes of data, making it challenging to derive meaningful insights efficiently.
Innovation Solution
The method involves generating a lowest level data layer that serves multiple higher level data layers, allowing for rapid access and processing of queries by determining the highest level data layer from which a response can be generated, thereby reducing response times and optimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a BI system processes queries on large volumes of raw data, then the amount of accessible data increases, but query processing time increases
Solution Approach 1:
The system pre-generates multiple levels of aggregated datasets before queries are executed. The first level dataset aggregates data from raw data, the second level aggregates from the first level, and so on. When a query arrives, the system can immediately query these pre-computed datasets without processing raw data, thus reducing query processing time while maintaining access to large volumes of data.
Solution Approach 2:
The system divides the large volume of raw data into multiple hierarchical levels of datasets. Each level represents a different degree of aggregation, allowing the system to segment the data processing task into manageable chunks that can be queried efficiently at appropriate levels rather than processing all raw data for every query.
2Quantity of substance
If a BI system accesses lower level data layers, then comprehensive data is available, but access speed decreases
Solution Approach 1:
The system introduces a hierarchical dimension to data organization, creating multiple levels of aggregation (first level, second level, etc.). This allows the system to navigate different dimensions of data granularity - querying higher levels for fast access to aggregated insights and only descending to lower levels when detailed raw data is specifically needed, thus balancing data availability with access speed.
Data Source
AI summary
A system and method for efficiently responding to a query. The method comprises generating a lowest level data layer, wherein the lowest level data layer is a common dataset that can be served by a plurality of higher level data layers; generating, based on the lowest level data layer, at least one dataset, wherein each generated dataset is one of the plurality of higher level data layers, wherein each higher level data layer is accessed more rapidly than all lower level data layers; searching, in at least one of the generated data layers, for data needed to generate a response to the query; determining, based on the search, at least one data layer from which the response can be generated, wherein the determined at least one data layer includes the highest level data layer from which the response can be generated; and generating, based on data of the determined at least one data layer, a response to the query.