Distributed Indexing Architecture for Big Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional big data systems face challenges in creating a distributed indexing system across multiple machines due to memory resource consumption and increased search times as the indexing system grows, leading to degraded performance.

Innovation Solution

A distributed indexing architecture that partitions and distributes index tables among multiple devices, enabling parallel searches to reduce search times and memory usage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a conventional index system is implemented in each machine, then the indexing system can be distributed across multiple machines, but memory resources are consumed excessively because indexing systems are large data structures

Engineering Contradiction:

Improvedistributed indexing capabilityVSAvoidmemory resource consumption

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The patent divides the large index system into multiple smaller index tables that can be distributed across different machines. Each machine stores only a portion of the index data rather than complete copies, reducing memory consumption while maintaining distributed querying capability through coordinated access to segmented index tables

Inventive Principle:

Principle #1Segmentation

2Quantity of substance

If the index system grows to support more data, then the indexing system can handle larger data sets, but search time increases linearly due to deeper binary tree structures

Engineering Contradiction:

Improvedata set sizeVSAvoidsearch time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent segments the large index into multiple smaller index tables, allowing parallel search operations across different segments. This reduces the depth of individual binary trees and enables concurrent processing, thereby reducing overall search time while supporting larger data sets

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements overlapping index tables where certain index entries are replicated across multiple tables. This allows search operations to proceed in parallel without gaps, maintaining continuous useful action across all index tables simultaneously rather than sequential access

Inventive Principle:

Principle #20Continuity of useful action

3Reliability

If the index system is updated frequently to maintain data consistency, then the indexing system remains synchronized with stored data, but bandwidth and processing resources are consumed due to constant communications

Engineering Contradiction:

Improvedata consistencyVSAvoidbandwidth and processing resource consumption

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent allows different index tables to be updated independently based on local data changes rather than requiring simultaneous updates across all tables. Each machine can update its local index tables autonomously, reducing communication overhead while maintaining overall system consistency through the distributed architecture

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS10990574B2Distributed indexing architecture for databases

Publication Date: 2021.04.27 BANK OF AMERICA CORP

AI summary

A data storage device that includes a memory operable to store a data table and an indexing engine implemented by a processor. The indexing engine is configured to receive data and to store the data in the data table. The indexing engine is further configured to determine an index key and data location information for the stored data and to determine a set of index table references based on the index key. Each index table reference identifies an index table that links index keys with data location information. The indexing engine is further configured to identify a set of index tables corresponding with the set of index table references and to store the index key and the data location information for the stored data in one or more index tables from the set of index tables.