Hierarchical Data Column Storage for Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing methods, such as the map-reduce framework, face high disk I/O and communication costs due to the hierarchical nature of data, where entire records are fetched and processed even though only a few attributes are relevant to the query, leading to wasteful handling of useless data.
Innovation Solution
Hierarchical data is stored in column data structures, allowing only relevant data to be handled and processed, with each attribute stored separately, enabling the creation of intermediate data for targeted query execution, reducing the amount of data processed and communicated.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hierarchical data is stored in traditional row-based format and processed using map-reduce framework, then data can be distributed across computing nodes, but entire records are fetched and processed even when only a few attributes are relevant, leading to high disk I/O costs and communication costs
Solution Approach 1:
The patent segments hierarchical data into flat records by breaking down nested structures into individual rows with columns representing different hierarchy levels. This segmentation allows the query processor to access only specific columns relevant to the query rather than fetching entire hierarchical records, thereby reducing disk I/O costs and improving query processing efficiency.
Solution Approach 2:
The patent extracts only the relevant attributes from hierarchical data structures and stores them as separate columns in a flat table format. When a query is executed, only the extracted relevant columns are accessed and processed, eliminating the need to handle useless data and reducing communication costs across computing nodes.
2Productivity
If hierarchical data is stored in traditional row-based format, then data structure is simple to implement, but useless data is handled and processed along with relevant data, leading to wasteful processing
Solution Approach 1:
The patent segments hierarchical data into flat records with distinct columns for different attributes. This segmentation enables the system to process only the specific columns relevant to each query, eliminating the wasteful handling of unrelated data and improving overall data processing efficiency.
Solution Approach 2:
The patent implements partial action by accessing and processing only the necessary columns required for query execution rather than processing entire records. This selective approach reduces computational overhead and eliminates wasteful processing of irrelevant data while maintaining query accuracy.
3Reliability
If entire hierarchical records are fetched for query processing, then complete data is available for analysis, but communication costs across computing nodes increase significantly
Solution Approach 1:
The patent extracts and stores relevant attributes from hierarchical records in a flat format, enabling the query processor to retrieve only the specific data needed for analysis. This extraction approach maintains data reliability for query purposes while significantly reducing communication costs by avoiding the transmission of unnecessary data across computing nodes.
Solution Approach 2:
The patent segments hierarchical data into independent rows and columns, allowing the system to fetch and communicate only the specific columns relevant to the query. This segmentation preserves data completeness for analysis while minimizing communication overhead by excluding irrelevant data from transmission.
Data Source
AI summary
An embodiment provides intermediate data derived in the form of column stores which are in turn based on hierarchical data stores. This intermediate data represents a reduced subset of data matched appropriately to a query (or modified query) such that the amount of data handled in a query processing task on large data is greatly reduced. An embodiment may appropriately choose column data stores and/or modify queries in order leverage parallelization techniques such as map-reduce in order to query large data. The result is the ability to query large data stores in parallel while reducing the amount of data that must be handled.