Hierarchical Data Skipping via Aggregated Sketches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems in hierarchically organized computing environments face inefficiencies in querying data due to centralized storage, which leads to increased resource usage and security concerns, especially as the number of endpoints increases, and existing techniques require querying all nodes, consuming network resources and potentially compromising data security by centralizing data.
Innovation Solution
Implementing a method where processor units in a hierarchically organized computing system generate leaf node data sketches that summarize attributes of data at leaf nodes, aggregate these sketches at intermediate nodes, and retain them to facilitate data skipping, allowing queries to be directed only to nodes with matching data, reducing unnecessary processing and communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is centralized in one or a few locations for querying, then query processing can be simplified, but network resource usage increases and data security is compromised
Solution Approach 1:
The patent segments data storage across multiple distributed nodes rather than centralizing it. Each node maintains local data copies, and queries are routed to specific nodes based on data location information stored at higher hierarchy levels. This segmentation reduces unnecessary network traffic while maintaining query processing capability.
Solution Approach 2:
The patent introduces an intermediary data structure (data sketch or index) at intermediate hierarchy levels that acts as a mediator between queries and leaf nodes. This intermediary contains location information about data without requiring actual data transfer, enabling efficient query routing to relevant nodes only.
2Reliability
If all nodes are queried to ensure data completeness, then data retrieval reliability is improved, but processing time and network resources increase
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing location information (data sketches) at intermediate hierarchy levels that indicate which leaf nodes contain data matching certain criteria. This preliminary indexing allows queries to skip irrelevant nodes without sacrificing completeness, as the index is maintained to reflect actual data distribution.
Solution Approach 2:
The patent implements skipping by using data sketches to identify and skip nodes that definitely do not contain matching data. Queries can rush through the hierarchy by jumping directly to relevant leaf nodes based on the sketch information, avoiding unnecessary processing at intermediate nodes and eliminating wasted time on irrelevant data.
3Object-affected harmful factors
If data is decentralized across many nodes, then data security is improved, but query processing complexity increases
Solution Approach 1:
The patent maintains data segmentation across distributed nodes for security, while introducing a hierarchical structure with intermediate levels that segment the query processing task. Each intermediate level manages a portion of the search space, reducing the complexity burden on individual nodes and enabling coordinated query processing across the distributed system.
Solution Approach 2:
The patent adds a hierarchical dimension to the distributed data structure, organizing nodes into multiple levels. This dimensional addition provides a new approach to query processing where instead of querying all nodes at once, the system can traverse the hierarchy dimensionally, reducing processing complexity while maintaining decentralized data storage for security.
4Adaptability or versatility
If more endpoints are added to the computing environment, then system capacity increases, but resource usage and security risks increase
Solution Approach 1:
The patent segments the system into a hierarchy with leaf nodes representing individual endpoints and intermediate nodes representing groups. This segmentation allows the system to scale by adding more leaf nodes without proportionally increasing query processing overhead, as intermediate nodes can aggregate and route queries efficiently to relevant leaf nodes only.
Data Source
AI summary
A computer implemented method performs data skipping in a hierarchically organized computing system. A group of processor units determines leaf node data sketches for data in leaf nodes in the hierarchically organized computing system. The leaf node data sketches summarize attributes of data in the leaf nodes. The group of processor units aggregates the leaf node data sketches at intermediate nodes in the hierarchically organized computing system to form aggregated data sketches at the intermediate nodes and retains data sketches received at the intermediate nodes from a group of child nodes to form retained data sketches. The retained data sketches are one of leaf node data sketches and the aggregated data sketches. The group of processor units searches the data using the retained data sketches and the data skipping within the hierarchically organized computing system in response to queries made to the intermediate nodes in the hierarchically organized computing system.