Secondary Indexing via Sorted Replicas in Distributed File Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data stores face limitations in querying patterns due to lack of secondary indexes, leading to inefficient data retrieval and high latency, especially when querying on columns other than the row key, which requires full table scans and is not optimized for disk space efficiency.
Innovation Solution
The solution involves sorting replicas of master data at a file system level in a distributed file system to create secondary indexes on different columns, leveraging existing replication data for sorting, allowing for querying on different columns without full table scans and without requiring additional disk space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If secondary indexes are created on distributed data stores to enable efficient querying on columns other than row key, then querying performance is improved, but disk space consumption increases
Solution Approach 1:
The patent makes existing replication data serve dual purposes: maintaining data redundancy for fault tolerance and providing sorted secondary indexes for efficient querying. By sorting replicas on different columns and configuring them as secondary indexes, the system eliminates the need for separate index structures, allowing the same data copies to fulfill both protection and query optimization functions simultaneously.
Solution Approach 2:
The patent recovers value from existing replication data by reconfiguring sorted replicas as secondary indexes. Instead of treating replication data solely as redundant copies for fault tolerance, the system repurposes these sorted copies to provide efficient query paths, effectively recovering and utilizing otherwise wasted sorting effort and data organization.
2Ease of operation
If full table scans are performed to query on columns other than row key, then data retrieval is possible, but latency increases
Solution Approach 1:
The patent segments the monolithic data storage into multiple sorted replicas, each organized by different column values. This segmentation creates specialized query paths for different columns, allowing the system to route queries to the most appropriate sorted replica rather than scanning all data, thereby reducing query latency while maintaining full retrieval capability.
3Productivity
If traditional secondary indexes are implemented on distributed data stores, then querying efficiency on secondary columns is improved, but device complexity increases
Solution Approach 1:
The patent eliminates the need for separate index management systems by making existing replication data serve as secondary indexes. This universal usage of replication data for both fault tolerance and query optimization reduces device complexity by removing redundant index structures and their associated management overhead, while still providing efficient querying on secondary columns.
Data Source
AI summary
Embodiments for space-efficient secondary indexing on distributed data stores in a networked computing environment by a processor. Replicas of corresponding master data at a file system level of a distributed file system (DFS) are sorted for creating secondary indexes of the replicas on different columns of a database table, wherein existing replication data is leveraged for the sorting.