Distributed Storage Hash Table Lookup for Read Write Bottlenecks

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing distributed storage systems face performance bottlenecks due to resource-intensive sstable file merging and high storage costs associated with triple data copying, especially in large data sets with random read and write operations.

Innovation Solution

A data read and write method utilizing a hash table pre-loaded in memory to determine the storage location of shard data, allowing for efficient processing and reducing the need for binary search, while using a distributed storage system with 1.5 data copies to minimize storage costs and enhance reliability.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is stored in sstable files with binary search, then data organization is maintained, but system performance deteriorates due to resource-intensive merging processes

Engineering Contradiction:

Improvedata organizationVSAvoidsystem performance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent pre-loads the hash table into memory before processing requests, so that the mapping between data identifiers and storage locations is immediately available. This preliminary action eliminates the need for binary search during request processing, significantly improving system performance while maintaining data organization through the pre-established hash table structure.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent replaces the mechanical binary search process with a memory-based hash table lookup. Instead of performing sequential or binary searches through sorted files, the system uses hash function computations and direct memory access to retrieve storage locations, substituting a computationally intensive mechanical search with a faster computational approach.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If sstable files are merged and ordered frequently, then data organization is maintained, but resource consumption increases leading to system bottleneck

Engineering Contradiction:

Improvedata organizationVSAvoidresource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The hash table is pre-loaded into memory with all the mapping information before the system processes requests. This preliminary organization of data identifiers and their corresponding storage locations eliminates the need for frequent merging and re-ordering operations, maintaining data organization while significantly reducing resource consumption.

Inventive Principle:

Principle #10Preliminary action

3Reliability

If data is stored in three copies, then reliability is improved, but storage costs increase

Engineering Contradiction:

Improvedata redundancyVSAvoidstorage costs

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent changes the redundancy parameter from 3 copies to 1.5 copies by implementing a distributed storage system where data is replicated across multiple shard servers. Each shard server maintains a hash table with data identifier to storage location mappings, providing the necessary redundancy for reliability while reducing the overall storage overhead compared to traditional triple copying.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11537659B2Method for reading and writing data and distributed storage system

Publication Date: 2022.12.27 BEIJING BAIDU NETCOM SCI & TECH CO LTD

AI summary

The present application discloses a data read and write method and a distributed storage system. A specific implementation of the method includes: receiving, from a client, by a shard server, a processing request on shard data, the processing request comprising a data identifier of the shard data; processing the processing request based on a hash table pre-loaded in a memory and indicating a correspondence between the data identifier of the shard data and a data index to obtain a processing result; and sending the processing result to the client.