Committed Throughput Allocation for Auto-Partitioned Data Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage services face challenges in providing seamless scaling, predictable performance, and cost-effective solutions for large-scale, non-relational databases, especially in managing massive tables without imposing maximum size or throughput limits, while maintaining durability, availability, and supporting flexible schema and consistency models.
Innovation Solution
A web services-based data storage service that supports automatic live repartitioning and scaling in response to anomalies, offers flexible schema, multiple indexing options, and consistency models, and provides an API for seamless client interaction, ensuring low latency and predictable performance without requiring client-side partitioning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple machines for security and availability, then reliability and availability are improved, but system complexity and difficulty of management increase
Solution Approach 1:
The patent implements automatic partitioning where the system autonomously divides large tables into smaller partitions and distributes them across multiple machines without requiring manual intervention. The system self-manages the complexity of distribution, balancing data across nodes based on size and accessibility requirements, thereby improving availability while hiding the complexity from users.
Solution Approach 2:
The patent segments large tables into smaller partitions that can be independently managed and distributed across multiple machines. This segmentation allows the system to maintain reliability through distribution while reducing the complexity of managing individual data units, as each partition can be handled independently by the automatic partitioning mechanism.
2Adaptability or versatility
If automatic live repartitioning and scaling is implemented, then scalability and adaptability are improved, but device complexity and processing overhead increase
Solution Approach 1:
The patent implements automatic partitioning that monitors table size and accessibility patterns, providing feedback to trigger repartitioning operations when thresholds are exceeded. This feedback mechanism enables scalable adaptation to changing data volumes while managing processing overhead by activating repartitioning only when necessary, rather than continuously.
Solution Approach 2:
The patent performs preliminary partitioning actions by pre-dividing tables into manageable partitions and pre-configuring the automatic partitioning mechanism. This preliminary setup allows the system to handle scalability requirements without incurring high processing overhead during actual data operations, as the repartitioning infrastructure is already in place and ready to execute when needed.
3Adaptability or versatility
If no maximum size or throughput limits are imposed, then versatility and adaptability are improved, but system reliability and performance predictability worsen
Solution Approach 1:
The patent implements dynamic automatic partitioning that adapts to varying data sizes and throughput requirements without imposing fixed limits. The system dynamically creates, merges, and redistributes partitions based on real-time conditions, maintaining performance predictability through consistent automated management while preserving flexibility to handle any data volume.
Solution Approach 2:
The patent changes system parameters dynamically by adjusting partition sizes, numbers, and distribution strategies based on data volume and access patterns. This parameter change approach allows the system to maintain reliability and performance predictability across different scales while preserving versatility, as the automatic partitioning mechanism adapts its parameters to match actual workloads.
4Ease of operation
If client-side partitioning is required, then control and management are improved, but ease of operation and complexity reduce
Solution Approach 1:
The patent implements server-side automatic partitioning that eliminates the need for client-side partitioning operations. The system performs all partitioning, distribution, and balancing operations autonomously, thereby simplifying client operations to basic data storage and retrieval while maintaining the complexity management on the server side through automated mechanisms.
Data Source
AI summary
A system that implements a scaleable data storage service may maintain tables in a data store on behalf of storage service clients. The service may maintain data in partitions stored on respective computing nodes in the system. The service may support multiple throughput models, including a committed throughput model and a best effort throughput model. A service request to create a table may specify that requests directed to the table should be serviced under a committed throughput model and may specify the committed throughput level in terms of logical service request units. The service may reserve low-latency storage and other resources sufficient to meet the specified committed throughput level. A client/user may request a modification to the committed throughput level in anticipation of workload changes, such as an increase or decrease in traffic or data volume. In response, the system may increase or decrease the resources reserved for the table.