Secondary Index Progress Tracking in Distributed Data Stores

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Creating secondary indexes in distributed database systems is resource-intensive and can lead to inconsistencies, especially when dealing with large datasets, as it requires maintaining data consistency across multiple partitions while allowing the system to remain online for access requests, which complicates determining the progress of indexing operations.

Innovation Solution

Implementing a method to determine the creation progress point of a secondary index across distributed data store partitions by averaging completion percentages or using weighted averages based on item counts, allowing for responsive actions to optimize indexing speed and ensuring accurate progress indication.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If secondary index creation is performed on large distributed datasets, then indexing completeness is improved, but system resource consumption increases and consistency maintenance becomes complex

Engineering Contradiction:

Improveindexing completenessVSAvoidsystem complexity

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent divides the distributed table into multiple partitions and creates secondary indexes for each partition independently. This segmentation allows the indexing process to be distributed across multiple nodes, reducing the complexity burden on any single node while achieving complete indexing across the entire distributed dataset.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces progress tracking mechanisms that monitor and report indexing status before the process is fully complete. This preliminary action enables the system to provide visibility into indexing progress and allows for optimization decisions to be made during the indexing process, rather than waiting for completion.

Inventive Principle:

Principle #10Preliminary action

2Manufacturing precision

If secondary index creation is performed on large distributed datasets, then indexing completeness is improved, but resource consumption increases

Engineering Contradiction:

Improveindexing completenessVSAvoidresource consumption

Core Design Contradiction:

Manufacturing precisionVSUse of energy by moving object

Solution Approach 1:

By segmenting the indexing process into partition-level operations, the patent distributes resource consumption across multiple nodes rather than concentrating it on a single node. This allows the system to achieve complete indexing while preventing any single node from being overwhelmed by excessive resource demands.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent enables the system to perform indexing operations in stages, completing indexes for individual partitions rather than requiring all partitions to be indexed simultaneously. This partial action approach allows the system to achieve functional completeness while managing resource consumption at sustainable levels.

Inventive Principle:

Principle #16Partial or excessive action

3Reliability

If the system remains online during indexing operations, then service availability is maintained, but determining indexing progress becomes difficult

Engineering Contradiction:

Improveservice availabilityVSAvoidprogress determination difficulty

Core Design Contradiction:

ReliabilityVSDifficulty of detecting and measuring

Solution Approach 1:

The patent implements progress tracking mechanisms that continuously monitor and report the status of secondary index creation across partitions. This feedback system provides visibility into indexing progress while the system remains online, allowing users to monitor completion status without requiring the system to go offline.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent creates a unified progress tracking mechanism that works across all partitions simultaneously. This universal approach allows the system to provide consistent progress information regardless of which partitions are being indexed, making progress determination straightforward even while the system continues to serve requests.

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Adaptability or versatility

If secondary indexes are created across distributed partitions, then data accessibility is improved, but consistency between table and index becomes challenging

Engineering Contradiction:

Improvedata accessibilityVSAvoiddata consistency

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent maintains consistency by treating each partition as an independent unit for indexing purposes. By ensuring consistency within each partition separately rather than attempting to manage consistency across the entire distributed table at once, the system achieves reliable data consistency while maintaining the accessibility benefits of distributed indexing.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11327937B1Determining indexing progress for a table in a distributed data store

Publication Date: 2022.05.10 AMAZON TECH INC

AI summary

A distributed data storage system may implement determining indexing progress for a table stored in the distributed data storage system. A table may be stored in multiple table partitions. When a secondary index is created for the table, each partition may independently index the items stored within the table partition in order to identify those items in the partition that should be stored in the secondary index. During creation of the secondary index, creation progress points that indicate the progress of the indexing across the table partitions may be determined. The creation progress points may be provided via a programmatic, textual, or graphical interface to the distributed data storage system.