Data Grid Topology Change During Query Iteration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data grids using the map/reduce application struggle to adjust and adapt to topology changes, such as node crashes, additions, or rebalancing, leading to issues like out-of-date data sets, incorrect information, and delays in adding new nodes, which affect data processing and storage efficiency.
Innovation Solution
Implementing a method that allows for topology changes and data rebalancing during query iterations, using a coordinator to manage node communications and rebalance data, ensuring up-to-date results and variable memory capacity, thereby enabling efficient data access and search operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional map/reduce application is used for querying data in a data grid, then distributed processing can be provided for large data sets, but the data grid cannot adjust or adapt to topology changes such as node crashes, nodes joining or leaving, or balancing of data between nodes
Solution Approach 1:
The patent implements dynamic topology management by allowing the data grid to detect and respond to topology changes (node additions, removals, crashes) during query execution. The system dynamically adjusts the query plan and redistributes data segments across available nodes, enabling the grid to adapt its structure while maintaining processing reliability through continuous monitoring and reconfiguration capabilities
2Quantity of substance
If data grid size is adjusted to vary with data usage, then storage capacity can be optimized, but topology changes during query iterations cause out-of-date data sets and incorrect information
Solution Approach 1:
The patent implements a feedback mechanism where the system continuously monitors topology changes during query execution and uses this information to update the query plan. When data segments are added or removed from nodes during iteration, the system detects these changes and recalculates the query results, ensuring that the final data set reflects the current grid state and maintains accuracy despite dynamic storage adjustments
Solution Approach 2:
The patent performs preliminary actions by establishing a initial query plan before topology changes occur, then systematically updating this plan as changes are detected. The system proactively identifies when topology changes affect the current query execution and preemptively adjusts the data retrieval and aggregation process to maintain data accuracy throughout the iteration
3Productivity
If new nodes are added to the data grid during query execution, then variable memory capacity is achieved, but traditional methods cause delays in adding new nodes
Solution Approach 1:
The patent enables dynamic node addition during query execution by implementing a flexible architecture where new nodes can be integrated into the data grid without stopping or pausing ongoing queries. The system dynamically assigns data segments to new nodes and adjusts the query plan in real-time, allowing continuous operation and eliminating the need to wait for query completion before expanding storage capacity
Solution Approach 2:
The patent maintains continuity of useful action by ensuring that query processing continues uninterrupted when new nodes are added. The system seamlessly incorporates new nodes into the existing query execution flow, redistributing data segments and adjusting computation tasks without halting the overall processing operation, thus maintaining constant productivity while expanding capacity
4Adaptability or versatility
If topology changes occur during query execution, then adaptability is improved, but the master node may be unaware when data has been added to, removed from, or modified at a node
Solution Approach 1:
The patent implements comprehensive feedback mechanisms where nodes continuously report their data segment status, additions, removals, and modifications to the master node. This real-time feedback enables the master node to maintain an accurate view of the data grid topology and data locations, allowing it to dynamically adjust query plans and ensure no data is overlooked or processed from incorrect locations
Solution Approach 2:
The patent introduces an intermediary coordination layer that manages communication between nodes and the master node during topology changes. This intermediary tracks data segment movements and maintains a current map of data locations, serving as a mediator that ensures the master node has accurate information about where data resides even as the topology dynamically changes during query execution
Data Source
AI summary
A method of communicating data in a data grid. The method includes receiving a first search request defined by search criteria of a search query. The method includes communicating the first search request to search for data segments at the first node that match the search criteria. The method includes receiving one or more data segments matching the search criteria. The method includes receiving rebalance information. The method further includes identifying a second node storing rebalance data segments of the plurality of rebalance data segments based on the rebalance information. The method further includes communicating a second search request to search for the rebalance data segments matching the search criteria. The method further includes receiving one or more rebalance data segments matching the search criteria. The method further includes determining search result data using the data segments from the first node and the rebalance data segments from the second node.