Data Sharding by Surname for Database Search Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database search methods are inefficient and costly due to the increasing volume of data, leading to high compute and memory requirements, and the need for accurate and timely search results.
Innovation Solution
The implementation of a data-sharding system that facilitates automatic sharding of databases across multiple dimensions, including record type, year, and surname, to reduce the number of shards queried and optimize resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is stored in a single large database, then data accessibility is simplified, but search time and compute requirements increase significantly
Solution Approach 1:
The patent divides the large database into multiple smaller databases (shards) organized by surname. Each shard contains records for a specific surname range, allowing the system to search only relevant shards instead of scanning the entire database. This segmentation reduces search time while maintaining ease of access through the search engine that automatically routes queries to appropriate shards.
Solution Approach 2:
The patent introduces a new sharding dimension based on surnames, transforming the traditional single-dimension database structure into a multi-dimensional organization. Records are sorted and distributed across shards using surname as the key dimension, enabling parallel search operations across multiple shards simultaneously, thereby reducing overall search time.
2Speed
If the number of shards is increased to handle larger data volumes, then search speed improves, but system complexity and cost increase
Solution Approach 1:
The patent applies local quality by organizing shards based on surname distribution patterns. Each shard is tailored to handle specific surname ranges, and the search engine automatically determines which shards are relevant based on the query. This localized organization improves search speed for name-based queries while keeping the overall system manageable through automated routing.
Solution Approach 2:
The search engine performs self-service by automatically determining which shards should be queried based on the search criteria. When a user searches for a specific surname or name pattern, the search engine autonomously identifies and routes the query to the appropriate shards without requiring manual intervention or complex user-side processing, thereby improving speed while maintaining system simplicity.
3Ease of manufacture
If random distribution of records across shards is used, then data distribution is simplified, but the number of shards that must be queried increases
Solution Approach 1:
The patent applies preliminary action by pre-organizing records into shards based on surname before they are needed for search. The database is initially sorted and distributed across shards using surname as the key, so that when searches are performed, the relevant shards are already prepared and readily accessible. This preliminary organization eliminates the need for complex runtime distribution decisions and improves search efficiency.
4Measurement precision
If more shards are queried to ensure comprehensive search results, then search accuracy improves, but compute resources and time increase
Solution Approach 1:
The patent applies the extraction principle by isolating and querying only the specific shards that contain relevant records based on the search query. When a user searches for a particular surname or name pattern, the search engine extracts and queries only the shards that match the criteria, rather than scanning all shards. This selective extraction maintains search accuracy for name-based queries while significantly reducing compute resource consumption.
Data Source
AI summary
Data-sharding systems and/or methods for cost- and time-efficient record search are described. Data-sharding embodiments utilize a name-sharding dimension, optionally in combination with one or more additional dimensions such as record type and year, to reduce latency and reduce search-associated costs. The data-sharding systems and methods embodiments utilize an optimization algorithm to determine a distribution of records related to names. The optimization algorithm may use a three-character prefix for surnames in records to distribute shards across documents, with specific shards relating to no-name and multi-name records allocated.