Cosharding Index Tables to Reduce Distributed Retrieval Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed relational databases face inefficiencies in data retrieval due to the separation of tables across multiple database servers, requiring extensive distributed operations to locate and retrieve data, which can lead to increased complexity and processing time.
Innovation Solution
The method involves cosharding an index table based on the keys of a data table, ensuring that all index table rows corresponding to a single data table row are stored in a single split, allowing for local determination of data matches and reducing the need for distributed operations, and optionally sharding data tables with their index tables to improve data retrieval efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is separated across multiple database servers, then storage capacity and scalability are improved, but data retrieval efficiency deteriorates due to required distributed operations
Solution Approach 1:
The patent segments data into fixed-size batches and creates multiple versions of each batch, distributing them across different servers. This segmentation allows retrieval operations to work with complete batches locally rather than requiring distributed queries across servers, thus maintaining storage scalability while improving retrieval efficiency through localized access.
Solution Approach 2:
The patent creates multiple versions of data batches and distributes them to different servers, ensuring that each server contains complete data batches that can be retrieved independently. This local quality approach allows any server to potentially satisfy retrieval requests without requiring communication with other servers, resolving the contradiction between distributed storage and retrieval efficiency.
2Adaptability or versatility
If distributed operations are used to locate and retrieve data, then data access across multiple servers is enabled, but system complexity and processing time increase
Solution Approach 1:
The patent performs preliminary actions by pre-creating multiple versions of data batches and distributing them across servers before retrieval operations are needed. This advance preparation eliminates the need for complex distributed query operations at retrieval time, as the system can directly access pre-positioned data copies, thereby reducing both complexity and processing time while maintaining versatile data access.
3Quantity of substance
If traditional sharding is used to distribute data, then storage scalability is improved, but hotspot formation occurs leading to uneven server load
Solution Approach 1:
The patent makes each server multi-functional by distributing complete data batches to multiple servers rather than partitioning data across servers. Each server can independently handle retrieval operations for its stored batches, providing universal access capability across the distributed system. This approach maintains storage scalability while preventing hotspots, as load can be distributed to any server containing relevant data batches.
Data Source
AI summary
The technology relates to cosharding tables within a distributed storage system. A data table including one or more rows may be received. Each row in the data table may include an identifier key and pieces of data. Each piece of data in the data table may be indexed into individual rows of an index table, wherein each row in the index table includes data associated with the identifier key of the data table from which the piece of data in the respective row was indexed. The index table may be sharded into splits, wherein the sharding includes assigning each row of the index table into one of the splits based on the identifier key of the data table from which the piece of data in the respective row was indexed. The splits may be stored into two or more portions of the distributed storage system.