Distributed Key-Value Storage with Independent Object-Index Scaling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Key value store databases face challenges in handling advanced queries and data inconsistencies due to prioritizing speed over strong consistency and availability, especially as data traffic grows, necessitating a storage solution that supports query language-based search and can scale horizontally indefinitely.

Innovation Solution

Implementing a scalable key value storage system in a distributed storage system that separates object storage from index storage, allowing both to scale independently, uses consistent hashing for partition assignment, and employs change data capture to update indexes dynamically, with large objects stored in external storage and small pointers in object storage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If key value stores prioritize speed over strong consistency, then data retrieval speed is improved, but data consistency and availability deteriorate

Engineering Contradiction:

Improvedata retrieval speedVSAvoiddata consistency

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The patent segments the storage system into separate object storage and index storage components. Object storage handles data retrieval speed while index storage manages consistency through change data capture mechanisms. This segmentation allows each component to optimize for its specific function without compromising the other.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary indexing layer that mediates between object storage and query operations. The index storage acts as a buffer that maintains consistency information separately, allowing fast object retrieval while ensuring data integrity through the indexing mechanism.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If key value stores scale horizontally, then data capacity and performance are improved, but system complexity increases

Engineering Contradiction:

Improvedata capacityVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent divides the distributed storage system into independent object storage nodes and index storage nodes. This segmentation allows horizontal scaling of data capacity without proportionally increasing system complexity, as each node operates semi-independently with standardized interfaces.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent creates universal indexing mechanisms that work across multiple storage nodes. The change data capture system provides a unified approach to maintaining consistency across the distributed system, reducing the complexity that would otherwise arise from node-specific consistency management.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Adaptability or versatility

If object storage and index storage are separated, then scalability is improved, but system complexity increases

Engineering Contradiction:

ImprovescalabilityVSAvoidstorage architecture complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent explicitly segments object storage and index storage into separate components that can scale independently. Object storage can be expanded for capacity while index storage scales for query performance, providing versatile scalability without requiring complex integrated scaling solutions.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements self-service mechanisms where the index storage automatically updates through change data capture from object storage. This automation reduces the operational complexity of managing separate storage systems, as the system self-maintains consistency without complex manual coordination.

Inventive Principle:

Principle #25Self-service

4Reliability

If change data capture is used to update indexes dynamically, then data consistency is improved, but processing overhead increases

Engineering Contradiction:

Improvedata consistencyVSAvoidprocessing overhead

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent applies preliminary action by capturing data changes at the source (object storage) before they need to be reflected in the index storage. This proactive approach to change capture ensures consistency is maintained with minimal processing overhead, as changes are recorded incrementally rather than requiring full index regeneration.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12450215B1Scalable key value storage in a distributed storage system

Publication Date: 2025.10.21 CROWDSTRIKE

AI summary

Techniques for implementing a scalable key value storage in a distributed storage separate the storage of the collection of data objects from the storage of the index corresponding to the collection. According to an implementation, a database service may receive a request to create a collection of data objects in an object storage. A schema may be specified for the collection. The database service may generate a unique identifier (ID) corresponding to the collection and create the collection in the object storage. The objects in the collection and the associated schema may be further written to an object table and a schema table, respectively. The database service may further validate the schema and extract one or more indexable fields from the schema. The database service may send a request to a database search service to create an index for the collection in an index storage.