Distributed Data Set Indexing via Super Cell Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently generating and managing indexes for large data sets across multiple node devices, which hinders fast data retrieval and processing in distributed systems, especially when dealing with vast and dynamic data sets.
Innovation Solution
The solution involves dividing data into super cells and data cells, generating super cell indexes and cell indexes that include unique values, duplicates, and range indications, allowing for efficient data access and search operations across multiple node devices through parallel processing and distributed indexing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored across multiple node devices in a distributed system, then storage capacity and system scalability are improved, but indexing efficiency and data retrieval speed deteriorate due to the need to coordinate across multiple devices
Solution Approach 1:
The patent segments the data set into multiple data partitions distributed across different node devices, with each partition having its own local index. This segmentation allows the system to maintain high storage capacity while improving indexing efficiency by limiting index operations to local partitions rather than requiring coordination across the entire distributed system.
Solution Approach 2:
The patent introduces a coordinator node that acts as an intermediary between query requests and distributed data partitions. The coordinator receives search queries, determines which partitions need to be searched based on index information, and collects results from relevant nodes, thereby mediating the complexity of distributed indexing operations.
2Device complexity
If a traditional centralized indexing approach is used, then indexing simplicity is maintained, but system scalability and data distribution flexibility deteriorate
Solution Approach 1:
The indexing function is segmented and distributed to individual node devices, with each node maintaining a local index for its data partition. This eliminates the need for a centralized indexing system while preserving simplicity at the local level, and simultaneously enables system scalability through the distributed architecture.
Solution Approach 2:
Each node device in the distributed system performs multiple functions: it stores data partitions, maintains local indexes, processes search queries for its partitions, and communicates with the coordinator. This multi-functionality eliminates the need for separate centralized indexing components while maintaining system scalability.
3Reliability
If data partitions are distributed across multiple nodes, then fault tolerance is improved, but coordination overhead and search performance deteriorate
Solution Approach 1:
The patent performs preliminary indexing of data partitions at the time of data insertion or update, creating local indexes that enable rapid search operations. This preliminary action ensures that when search queries arrive, the system can immediately utilize pre-built local indexes without requiring coordination to build indexes on-demand, thereby reducing search time while maintaining fault tolerance through distribution.
Solution Approach 2:
The patent extracts index information from the distributed data partitions and makes it available to the coordinator node. The coordinator uses this extracted index information to determine which partitions need to be searched, eliminating the need for full data transfers and reducing coordination overhead while maintaining search performance.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An apparatus including a processor to receive search criteria including a data value for a search within a data field; in response to the receipt of the query instructions, and for each data cell within a super cell, perform the specified search by comparing the data value to ranges of values indicated in a corresponding cell index to determine whether the data cell includes a data record meeting the search criteria, and in response to a determination that the data cell includes such a data record, use a unique values index in the cell index to search the data records of the data cell to identify one or more data records meeting the search criteria; and in response to identifying at least one data record meeting the search criteria, provide an indication that at least the data cell includes at least one data record meeting the search criteria.