Hybrid Hash Data Placement Algorithm for Distributed Query Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed query processing systems face challenges in data placement, including centralized data mapping bottlenecks, failure resilience, and high data lookup latency due to the limitations of existing centralized and decentralized algorithms.

Innovation Solution

A hybrid data placement algorithm using two or more hash functions to identify a destination node for data placement, ensuring load balance and minimal storage and computational resources, while maintaining failure resilience by iteratively selecting available nodes and utilizing an exception map for unavailable nodes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If centralized data mapping is used, then data placement decisions can be made efficiently, but system bottleneck occurs and failure resilience is compromised

Engineering Contradiction:

Improvedata placement decision efficiencyVSAvoidfailure resilience

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent segments the centralized data mapping function by introducing per-node data maps that store only local data placement information. Each node maintains its own data map for data it stores, eliminating the need for a single centralized data map. This segmentation distributes the mapping function across nodes, reducing the bottleneck at any single node while maintaining placement decision efficiency locally.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a data placement service as an intermediary component that mediates between data placement requests and node selection. This service uses a placement algorithm to determine destination nodes and manages the data placement process, distributing the mapping function and eliminating the need for a centralized data map while maintaining efficient placement decisions.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Ease of operation

If centralized data mapping is used, then data placement can be coordinated, but storage space requirement increases at the central node

Engineering Contradiction:

Improvedata placement coordinationVSAvoidstorage space at central node

Core Design Contradiction:

Ease of operationVSQuantity of substance

Solution Approach 1:

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Instead of having a single centralized data map, the patent implements multiple copies of data mapping information distributed across different nodes. Each node has a local data map containing placement information for data stored at that node. This copying approach eliminates the need for a large centralized data map while maintaining the ability to coordinate data placement decisions through the data placement service.

Inventive Principle:

Principle #26Copying

3Reliability

If decentralized data mapping is used, then failure resilience is improved, but data lookup latency increases

Engineering Contradiction:

Improvefailure resilienceVSAvoiddata lookup latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent implements feedback mechanisms where nodes report their availability status and data placement information to the data placement service. The service uses this feedback to make informed placement decisions and update data maps, enabling efficient data lookup while maintaining failure resilience through distributed mapping.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent performs preliminary actions by pre-computing and caching data placement decisions and node availability information in per-node data maps. This allows nodes to quickly determine data locations and availability without performing complex decentralized routing during data lookup operations, reducing latency while maintaining failure resilience.

Inventive Principle:

Principle #10Preliminary action

4Productivity

If hash functions are used for data placement, then load balancing is achieved, but computational resources are required

Engineering Contradiction:

Improveload balancing efficiencyVSAvoidcomputational resources

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent changes the parameters of the placement algorithm by using hash functions that map data identifiers to node identifiers in a way that achieves load balancing. The algorithm uses parameters such as data identifier hashing and node identifier mapping to distribute data evenly across nodes while using minimal computational resources for the placement decisions.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS9842148B2Method for failure-resilient data placement in a distributed query processing system

Publication Date: 2017.12.12 ORACLE INT CORP

AI summary

Herein is described a data placement scheme for a distributed query processing systems that achieves load balance amongst the nodes of the system. To identify a node on which to place particular data, a supervisor node performs a placement algorithm over the particular data's identifier, where the placement algorithm utilizes two or more hash functions. The supervisor node runs the placement algorithm until a destination node is identified that is available to store the data, or the supervisor node has run the placement algorithm an established number of times. If no available node is identified using the placement algorithm, then an available destination node is identified for the particular data and information identifying the data and the selected destination node is included in an exception map. Most data may be located by any node in the system based on the node performing the placement algorithm for the required data.